DEV Community: RL Loz

Optimizing Bulk Product Uploads: Handling Embedded Images and Partial Success in Go

RL Loz — Fri, 12 Jun 2026 16:01:32 +0000

The Concept

When designing bulk data-ingestion pipelines, engineering teams often face a tension between strict data integrity and user experience. A naive bulk-import system treats the entire payload as a single transaction, if one row fails, the entire batch rolls back. While this guarantees data consistency, it creates a frustrating user bottleneck.

A mature bulk-import system should act as an intelligent filter: gracefully handling complex binary data (like embedded images), processing valid records in batches, and isolating failures so the user can fix and re-upload only what is broken.

The What

Our application features a critical tool allowing distributors to bulk-update products via XLSX spreadsheets. However, the initial implementation suffered from two major limitations:

Lack of Validation & Error Isolation: Any single row error would halt the entire file execution, failing the upload completely.
Missing Media Portability: Distributors opening new branches frequently requested the ability to automatically clone existing product images to the new branch profile via the spreadsheet.

The Why

We needed to redesign this pipeline to accomplish two goals:

Fault-Tolerant Processing: Ensure that valid product updates are successfully committed to the database, while invalid rows are isolated and surfaced back to the distributor for targeted correction.
Binary Data Extraction: Programmatically detect, extract, and reference product images embedded directly inside spreadsheet cells, automatically mapping them to the correct branch inventory records.

The How

1. File & Data Sanity Checking

Before any database operation occurs, the incoming payload undergoes strict sanitization:

MIME-Type Verification: The system strictly enforces that only valid .xlsx files are accepted.
Data Harmonization: Price fields are stripped of special characters, currency symbols, and whitespace, then safely parsed into clean numeric types.

2. Extracting Embedded Images with `excelize`

Reading images from an Excel file in Go using the excelize library requires a two-step coordinate extraction process, because calling a cell directly using standard string retrieval returns a #VALUE! error.

Step 1: Coordinate Mapping: We utilize GetPictureCells() to retrieve a list of all cell coordinates that actually contain image data.
Step 2: Binary Extraction: We loop through these coordinates and invoke GetPictures(). This function reads the raw cell data and returns a struct containing the raw File bytes, Extension, Format, and InsertType.
Step 3: Database Referencing: The raw bytes are saved to our storage server. We generate a unique, collision-resistant filename for each image, inject this new filename reference back into our internal representation of the spreadsheet rows, and prepare it for database persistence.

3. Batch Processing and Partial-Success Strategy

To handle high concurrency and prevent the database from being bottlenecked by row-by-row INSERT queries, we implemented a batch-processing algorithm designed around pre-fetching and comparing:

[Distributor Uploads XLSX] 
           │
           ▼
[Extract all SKUs & Names] 
           │
           ▼
[Single Query: Select Existing from DB] 
           │
           ▼
[Comparison] ───────────────────┐
   │                            │
   ▼ (No Conflicts)             ▼ (Conflicts)
[Batch Insert Valid]     [Collect Errors]
   │                            │
   ▼                            ▼
[Update Inventory & Stocks]  [Return Error Report to User]

Pre-fetching Constraints: The system loops through the spreadsheet memory cache to compile a list of all SKUs and product names.
Bulk Conflict Checking: We hit the database once using this compiled list to pull all existing records matching those SKUs or names, creating an in-memory lookup map of existingProducts.
The Comparison: We compare the incoming spreadsheet rows against our database lookup map:
Valid Products: Rows without conflicts are grouped into a Batch Insert transaction. Successful inserts automatically trigger downstream events to generate inventory records and historical stock movements.
Invalid Products: Rows with data anomalies or conflicts are skipped, collected into an error array, and returned to the distributor at the end of the execution.

The Trade-offs

Pros:

Drastically Improved UX: Distributors no longer have to guess which row broke the upload; they receive a clean report of what failed and can keep working seamlessly.
Reduced Database I/O: Moving away from a "loop-and-insert" pattern to a single pre-fetch query followed by a batch insert significantly reduces database connection overhead and lock contention.

Cons / Risks:

Memory Overhead: Holding an entire spreadsheet's rows, image bytes, and a database lookup map in memory simultaneously increases the service's RAM footprint. For exceptionally large files, this risks hitting container memory limits.
Asynchronous Drift: Because valid rows are saved while invalid ones are rejected, the distributor's local spreadsheet becomes partially out-of-sync with the live database until they apply their fixes.

Layman's Terms: How It Works

Imagine a distributor hands you a giant box of 1,000 product sheets to log into your warehouse system.

Previously, if sheet #452 had a typo, you would throw your hands up, dump the entire box in the trash, and tell the distributor to fix the typo and bring all 1,000 sheets back. To make matters worse, if a sheet had a physical photo glued to it, you had no way of copying it to a new branch folder.

With our new system, we look at the whole box at once. We use a special tool to cleanly peel the photos off the paper and save them digitally. Then, we quickly check our computer to see which products look correct. If 950 products are perfect, we log them into the system immediately and update the warehouse shelves. For the 50 that had typos, we put them in a small folder and hand them back to the distributor saying, "These 950 are done. Just fix these 50 specific errors and give them back to us."

Conclusion

While bulk file handling with embedded media is a common hurdle in enterprise applications, approaching it with smart pre-fetching and a partial-success architecture creates a resilient system. If you are currently building a similar data-ingestion pipeline, implementing early cell-coordination mapping and in-memory comparison will save your users time and protect your database performance.

for code reference:

Refactoring Go API Unit Tests: Breaking Down the Testing Monolith

RL Loz — Fri, 12 Jun 2026 03:54:34 +0000

The Concept

When building backend APIs in Go, testing isn't just about code coverage, it's about long-term maintainability. As an application grows, a naive approach to unit testing can lead to "testing monoliths" where test setup, mocking, HTTP routing, and core business logic verification are jammed into a single, massive file.

To keep a codebase agile, your testing architecture must mirror the separation of concerns found in your production code. This means isolating the code that handles incoming network requests from the code that executes your core domain logic.

The What

Initially, the testing structure for the API lived entirely within a single, monolithic *_test.go file. Inside this file, everything was dumped together: Only service-layer assertions no HTTP request simulation, and handwritten mock structs for both the repository and service layers.

const UUID = "12345678-1234-5678-1234-567890123456"

type StockMovementTest struct {
    GetStockMovementsFunc         func(ctx context.Context, filters dto.BaseFilters) ([]models.StockMovement, int, error)
}

func (m *StockMovementTest) GetStockMovements(ctx context.Context, q dto.BaseFilters) ([]models.StockMovement, int, error) {
    if m.GetStockMovementsFunc != nil {
        return m.GetStockMovementsFunc(ctx, q)
    }

    return nil, 0, nil
}

func NewTestStockMovement(testStockMovementRepo *StockMovementTest) stock_movements.StockMovementService {
    emailSvc := mocks.NewTestEmailService()
    _, auditLogSvc := mocks.NewTestAuditService()
    awsSvc := mocks.NewTestAWSService()

    return stock_movements.NewStockMovementService(testStockMovementRepo, emailSvc, nil, auditLogSvc, awsSvc)
}

func TestGetStockMovements(t *testing.T) {
    testStockMovementRepo := &StockMovementTest{}
    testStockMovementSvc := NewTestStockMovement(testStockMovementRepo)

    t.Run("Get StockMovements", func(t *testing.T) {
        testStockMovementRepo.GetStockMovementsFunc = func(ctx context.Context, q dto.BaseFilters) ([]models.StockMovement, int, error) {
            CheckStockMovementQuery(t, q)
            return []models.StockMovement{{ProductID: 1}}, 1, nil
        }

        query := dto.Query{Search: "test", Limit: 10, Offset: 2, Sort: "change_amount ASC"}
        _, _, err := testStockMovementSvc.GetStockMovements(context.Background(), query)

        if err != nil {
            t.Errorf("Expected no error, got %v", err)
        }
    })
}

The Why

While a single-file approach works for small projects, it quickly becomes an anti-pattern due to two major pain points:

Code Bloating: A single file containing setup, table-driven test cases, and verbose mock definitions quickly grows to thousands of lines, making it incredibly difficult to navigate and maintain.
Circular Dependencies (Cyclic Imports): In Go, packages cannot import each other transitively. When mocks, domain logic, and HTTP transport layers are tightly coupled in tests, you risk hitting compilation errors because the boundaries between your database, service, and handler packages become blurred.

Separating these concerns ensures that your tests remain clean, compile quickly, and scale alongside your features.

The How

To resolve this, the monolithic test file was refactored into a modular structure by breaking it down into three distinct components: mock.go, *_service_test.go, and *_handler_test.go.

📂 package_test/
├── 📄 mock.go              # Shared mock definitions for service & repository layers
├── 📄 *_service_test.go    # Pure business logic unit tests
└── 📄 *_handler_test.go    # HTTP, routing, and request/response validation tests

// mock.go

type MockAuthService struct {
    //
}

func (m *MockAuthService) GenerateToken(userID int64) (string, error) {
    return "", nil
}

func (m *MockAuthService) ParseToken(token string) (int64, error) {
    return 0, nil
}

func (m *MockAuthService) Login(ctx context.Context, req LoginRequest) (string, error) {
    return "", nil
}

func (m *MockAuthService) Register(ctx context.Context, req RegisterRequest) error {
    return nil
}

type MockAuthRepository struct {
    GetUsernameOrEmailFunc func(ctx context.Context, username string) (*models.Users, error)
    RegisterFunc           func(ctx context.Context, user *models.Users) error
}

func (m *MockAuthRepository) Register(ctx context.Context, user *models.Users) error {
    if m.RegisterFunc != nil {
        return m.RegisterFunc(ctx, user)
    }

    return nil
}

func (m *MockAuthRepository) GetUsernameOrEmail(ctx context.Context, username string) (*models.Users, error) {
    if m.GetUsernameOrEmailFunc != nil {
        return m.GetUsernameOrEmailFunc(ctx, username)
    }

    return &models.Users{}, nil
}

// *_service_test.go

var testAuthRepo = &auth.MockAuthRepository{}
var testAuthSvc = NewTestAuth(testAuthRepo)

func NewTestAuth(testAuthRepo *auth.MockAuthRepository) auth.AuthService {
    config := &config.Config{JWTSecretKey: "airconcure_jwt_key"}

    return auth.NewAuthService(testAuthRepo, config)
}

func TestAuthServiceLogin(t *testing.T) {
    t.Run("login credentials", func(t *testing.T) {
        testAuthRepo.GetUsernameOrEmailFunc = func(ctx context.Context, username string) (*models.Users, error) {
            passwordHash, _ := bcrypt.GenerateFromPassword([]byte("!Abc1234"), bcrypt.DefaultCost)
            user := &models.Users{
                ID:       1,
                Username: "test_account",
                Email:    "test_account@local.com",
                Password: string(passwordHash),
            }

            return user, nil
        }

        req := auth.LoginRequest{
            Username: "rdev",
            Password: "!Abc1234",
        }

        _, err := testAuthSvc.Login(context.Background(), req)
        if err != nil {
            t.Errorf("Expected no error, got %v", err)
        }
    })
}

// *_handler_test.go

var testAuthHandler = auth.NewAuthHandler(&auth.MockAuthService{})

func TestAuthHandlerLogin(t *testing.T) {
    type loginRequest struct {
        Username string `json:"username"`
        Password string `json:"password"`
    }

    tests := []struct {
        name           string
        url            string
        requestBody    any
        expectedStatus int
    }{
        {
            name: "complete login credentials",
            url:  "/login",
            requestBody: loginRequest{
                Username: "rdev",
                Password: "!Abc1234",
            },
            expectedStatus: http.StatusOK,
        },
        {
            name: "login no password",
            url:  "/login",
            requestBody: loginRequest{
                Username: "rdev",
                Password: "",
            },
            expectedStatus: http.StatusBadRequest,
        },
        {
            name: "login password less than minimum required",
            url:  "/login",
            requestBody: loginRequest{
                Username: "rdev",
                Password: "1234",
            },
            expectedStatus: http.StatusBadRequest,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            ctx, w := helpers.SetupJSONTestContext(t, http.MethodPost, tt.url, tt.requestBody)

            testAuthHandler.Login(ctx)

            if w.Code != tt.expectedStatus {
                t.Errorf("Login() status = %d, resp = %v, want %d", w.Code, w.Body, tt.expectedStatus)
            }
        })
    }
}

1. Centralizing Mocks (`mock.go`)

Instead of rewriting or copy-pasting mock implementations across multiple test files, all repository and service mock structs are declared once inside a dedicated mock.go file within the package. This acts as a single source of truth for test doubles.

2. Testing the Business Logic (`*_service_test.go`)

This file focuses strictly on testing the "under-the-hood" domain logic. Because the database/repository layer is mocked out via mock.go, these tests run entirely in-memory and are incredibly fast. They validate:

Data validation rules and edge cases.
Handling of incorrect data types or malformed payloads.
Domain-specific error handling and state transitions.

3. Testing the HTTP Transport Layer (`*_handler_test.go`)

This file is dedicated to verifying how the API interacts with the outside world. It uses Go's net/http/httptest package to simulate client requests, utilizing the service mocks so it doesn't trigger actual business logic. These tests validate:

HTTP status codes (e.g., 200 OK, 400 Bad Request, 429 Too Many Requests).
JSON serialization and deserialization.
Query parameters, URL parameters, and headers.
Middleware execution, such as rate limiters and authentication checks.

The Trade-offs

Like any architectural decision, moving to a multi-file testing structure comes with balancing factors:

Pros:

High Scannability: Developers looking for a routing bug only need to open the handler test, while those fixing a calculation bug can go straight to the service test.
Reduced Friction: Isolating the mocks prevents cyclic imports, keeping the Go compiler happy.
Clear Boundaries: It enforces discipline, ensuring you don't accidentally test HTTP mechanics inside a business logic test.

Cons:

Boilerplate Overhead: Managing multiple files and orchestrating mocks requires slightly more upfront configuration and file management.
Mock Maintenance: If a service interface changes, the central mock.go file must be manually updated (unless you adopt a code-generation tool like mockery).

In Layman's Terms

Imagine you run a busy restaurant.

Originally, your test kitchen had one giant manual that crammed the cook’s recipes, the waiter’s serving rules, and cardboard cutouts of fake customers all onto the same page. It was crowded and confusing.

With this new approach, you split that manual into three neat folders:

The Prop Room (mock.go): This is where you keep all your cardboard cutouts (fake databases and fake chefs) so you can reuse them whenever you need to practice.
The Kitchen Manual (*_service_test.go): This is where you test the food itself. Does it taste right? Is it missing an ingredient? You don't care how the waiter delivers it; you just care that the recipe works perfectly.
The Dining Room Manual (*_handler_test.go): This is where you test the customer experience. Did the waiter smile? Was the bill calculated correctly? Is the host stopping too many people from rushing the door at once (rate limiting)? You don't care how the kitchen cooked the food here; you just care that the service at the table is seamless.

Conclusion

While software architecture preferences always vary depending on team conventions, decoupling your tests by responsibility is a proven strategy for Go applications. By isolating your HTTP logic from your business logic and centralizing your mocks, you eliminate code bloat, wipe out cyclic imports, and create a test suite that is easy to read, navigate, and maintain.

rdev-go-ddgen: Automating Domain Directory Boilerplate for Go Applications

RL Loz — Wed, 27 May 2026 00:55:43 +0000

When building robust backend APIs in Go, maintaining a consistent, clean architectural pattern across domain directories is essential for scalability. However, manually creating repetitive domain structures for every new resource introduces subtle human error.

The Why: Why Manual Boilerplate Fails
Copy-pasting existing files to scaffold a new domain (e.g., adding an orders domain next to a users domain) is a common but fragile practice. It is incredibly easy to overlook an import statement, miss a file, or forget to update a variable definition. As an API grows to dozens of domains, this manual approach becomes a major bottleneck that risks introducing compilation errors or silent bugs into your application.

The What: What is rdev-go-ddgen?
rdev-go-ddgen (Domain Directory Generator) is a lightweight, open-source Command Line Interface (CLI) tool designed to eliminate manual boilerplate creation for Go applications. It targets developers looking to instantly spin up predictable, structured domain directories without the risk of copy-paste mistakes.

While built out of a personal need while developing high-conformance Go APIs using Gin and Bun ORM, rdev-go-ddgen is completely engine-agnostic and serves as a universal foundational bootstrapper.

The How: Rapid Deployment
You can install rdev-go-ddgen globally and immediately integrate it into your development workflow with a single command:

go install github.com/XaiPhyr/rdev-go-ddgen@latest

To Initialize folder structure

rdev-go-ddgen init

To scaffold a completely new, structured domain directory, simply initialize the tool and use the domain flag (-d):

rdev-go-ddgen -d orders

This single command instantly generates a pristine directory containing five core architectural layers, fully stubbed with basic functions and type declarations:

handler.go – API routing and transport logic.
service.go – Core business logic boundaries.
repository.go – Database abstraction and data access.
types.go – Domain-specific data models and structs.
_test.go – Unit and integration test placeholders.

P.S. I hope this tool saves you some development time! rdev-go-ddgen is my first official contribution to the Go community, it’s a small step, but I'm incredibly excited to finally share something back with the ecosystem.

DEV Community: RL Loz

Optimizing Bulk Product Uploads: Handling Embedded Images and Partial Success in Go

The Concept

The What

The Why

The How

1. File & Data Sanity Checking

2. Extracting Embedded Images with excelize

3. Batch Processing and Partial-Success Strategy

The Trade-offs

Pros:

Cons / Risks:

Layman's Terms: How It Works

Conclusion

for code reference:

Refactoring Go API Unit Tests: Breaking Down the Testing Monolith

The Concept

The What

The Why

The How

1. Centralizing Mocks (mock.go)

2. Testing the Business Logic (*_service_test.go)

3. Testing the HTTP Transport Layer (*_handler_test.go)

The Trade-offs

In Layman's Terms

Conclusion

rdev-go-ddgen: Automating Domain Directory Boilerplate for Go Applications

2. Extracting Embedded Images with `excelize`

1. Centralizing Mocks (`mock.go`)

2. Testing the Business Logic (`*_service_test.go`)

3. Testing the HTTP Transport Layer (`*_handler_test.go`)