Home Projects Portfolio Dashboard Export PDF Log in

Achieving Atomic GitHub Syncs with Audit Logs and Data Preservation

On the landing project, which focuses on providing developers with insights into their activity, a critical component is the GitHub synchronization process. This process collects user data like commits and pull requests. However, as with any complex data synchronization, challenges arise: ensuring data consistency, preventing partial updates, and providing clear visibility into the sync's outcome.

This post dives into how we tackled these challenges by implementing atomic finalization steps, robust data merging strategies, and a comprehensive audit log system for our GitHub syncs.

The Challenge: Ensuring Data Consistency in Batch Syncs

Previously, our GitHub sync involved several batch jobs running in parallel. While efficient, this approach had a drawback: the user interface could update prematurely, showing an incomplete state. Furthermore, if an API call failed or returned nulls (e.g., due to a 403 error on an organizational repository), there was a risk of existing, valid deep sync data (like past commit counts) being inadvertently overwritten or reset to zero.

We needed a mechanism to ensure that the UI only reflected a fully completed sync and that valuable historical data was never lost during partial sync failures.

The Solution: Atomic Finalization with a Dedicated Job

To address the issue of partial UI updates, we introduced a dedicated FinalizeGitHubSyncJob. This job runs only after all individual batch synchronization tasks have completed. It consolidates critical final steps into a single, deferred operation:

  • Storing local participation data
  • Synchronizing metadata
  • Triggering the 'sync completed' notification

By deferring these actions, we ensure the UI remains consistent, only updating once all underlying data has been processed and is ready.

use App\Jobs\FinalizeGitHubSyncJob;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ProcessBatchGithubActivity implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function handle(): void
    {
        // ... process individual GitHub activity batches ...

        // After all batches are dispatched/processed, ensure finalization is queued
        if ($this->isLastBatch()) { // A hypothetical check for the final batch
            FinalizeGitHubSyncJob::dispatch($this->user)->onQueue('high');
        }
    }
}

This simple dispatch ensures that the FinalizeGitHubSyncJob acts as a gatekeeper, guaranteeing that the entire synchronization process completes atomically from the user's perspective.

Safeguarding Data Integrity: Merging Participation Data Intelligently

Preventing data loss, especially when API calls might return less data than previously stored, was crucial. We enhanced our GitHubActivityService with a mergeParticipationWithExisting method. This method intelligently compares incoming data with existing records. For metrics like user_commits or user_prs, it ensures that these counts never decrease during a sync. If a new API response returns nulls for a repository, our system preserves the previously recorded deep sync data, rather than overwriting it with an empty value.

namespace App\Services;

use App\Models\UserActivity;

class GitHubActivityService
{
    public function mergeParticipationWithExisting(UserActivity $existingActivity, array $newActivityData): UserActivity
    {
        // Example: Only update 'commits' if new data is greater or if it's the first time
        $existingActivity->commits = max(
            $existingActivity->commits ?? 0,
            $newActivityData['commits'] ?? 0
        );

        // Example: For PRs, only update if new data is present and valid
        if (isset($newActivityData['pull_requests']) && is_int($newActivityData['pull_requests'])) {
            $existingActivity->pull_requests = max(
                $existingActivity->pull_requests ?? 0,
                $newActivityData['pull_requests']
            );
        }

        // ... handle other fields with similar logic ...

        return $existingActivity;
    }
}

This logic ensures that our historical data remains robust against transient API issues or changes in repository access.

Observability with Audit Logs

To provide complete transparency and aid in debugging, we introduced a sync_audit_logs table and a corresponding read-only Filament resource. This audit log captures snapshots of project_participation before and after each synchronization run. It records the state of commits and PRs per repository, allowing us to visualize the delta (changes) over time. This offers an invaluable tool for verifying data accuracy and understanding the impact of each sync operation.

Combined with a dashboard action to reset a GitHub sync, complete with queued and completion notifications, we've built a highly observable and manageable synchronization feature.

The Takeaway

When dealing with complex batch processes and external APIs, prioritizing data consistency, integrity, and observability is paramount. Implementing atomic finalization steps ensures that users only see complete data. Strategically merging incoming data prevents accidental loss of valuable historical records. Finally, a robust audit logging system provides the transparency needed to trust your data and debug issues effectively. Always aim for atomic, durable, and observable data workflows.


Generated with Gitvlg.com

Achieving Atomic GitHub Syncs with Audit Logs and Data Preservation
Germanwithpants

Germanwithpants

Author

Share: