Capabilities | TechZavy

Accelerated MMCAP Infuse GPO Performance with Azure Databricks

The Challenge: Legacy Bottlenecks & Technical Debt

Performance Stalls: Aging legacy ETL packages required 1 hour 45 minutes to process critical Pharmacy Product and Pricing data.
Manual Friction: Analysts were burdened with manual FTP downloads and multi-step file transfers, data validations, and stage gates to trigger daily cycles to consolidate data
Infrastructure Gaps: On-premise latency and limited SQL "temp memory" created significant processing scale limits.

The TechZavy Solution: Spark-Driven "Lakehouse" Architecture

High-Performance Compute: Replaced "black box" ETL processes with Databricks and PySpark for distributed, parallel data processing.
Automated Lifecycle: Engineered Azure AI agents to coordinate pharmacy product file for procurement agencies after assessing data from NDC for pricing, formulary, dosage, and manufacturer information. Agents run FTP jobs and RAG validation in different source to profile, and transform raw data into procurement ready product files
Hardened Security: Integrated Azure Key Vault to eliminate exposed credentials and ensure federal-grade compliance.

The Business Impact: 5x Speed & Total Transparency

80% Reduction in Cycle Time: End-to-end processing plummeted from 105 minutes to just 20 minutes.
Eliminated Technical Debt: Modernized 30-year legacy logic into documented, auditable Python scripts.
Infinite Scalability: Transitioned from single-node SQL constraints to a distributed cluster architecture ready for national-scale growth.

Phase 1: Full Process Autonomy

Network Integration: Finalize static IP and private networking to eliminate the last manual file upload steps to Azure Storage.
Real-Time Triggers: Implement automated event notifications for the FDB (First Data Bank) server to initiate processing the moment data arrives.
Proactive Monitoring: Deploy automated health checks and "Smart Alerts" to notify stakeholders of successful cycle completions.

Phase 2: Workload Expansion

Pharmacy Sales Integration: Migrated high-volume Sales data into the Databricks Spark environment to match Product/Pricing speeds.
Weekly Program Refreshes: Transitioned the weekly refresh cycles from legacy SSIS to high-performance Python/Spark scripts.
Consolidated Reporting: Centralizing all Pharmacy workloads into the Lakehouse architecture for a "Single Source of Truth."

The Long-Term Vision: Data-Driven Agility

National Scalability: Leveraging distributed clusters to handle increasing GPO member volume without infrastructure strain.
Predictive Analytics: Utilizing the Lakehouse foundation to enable future AI/ML modeling on pricing trends and procurement cycles.
Audit-Ready Governance: Maintaining a 100% transparent, documented code base for federal and state compliance audits.

Agentic AI