Chunk Iterator

Registered by Daniel Nichter

Chunking is a fundamental and crucial operation for pt-archiver, pt-table-checksum, pt-table-sync, and other potential tools that access rows. The current code is fragmented and very complex, and no two tools chunk the same (they generate the same chunks but how they get those chunks internally varies widely). There are a few of issues inherited from Maatkit about chunking: http://code.google.com/p/maatkit/issues/list?&q=tag%3Dchunking The list is probably a little longer than that because it doesn't include char chunking and considerations that are sensitive to chunking like --resume.

I propose we write a chunk iterator that provides a simple interface to the complexities of chunking. Ideally, the chunk iter would do all the work behinds the scenes, but with hints, so a tool can simply $chunk->next(). At the tool level, we just want chunks regardless of table size, zero chunks, chunk index or columns, etc.

Unifying the chunking code would make fixing chunking bugs easier since all tools would be using the same chunking interface. A unified chunk iter would also allow us to add on other features in a nice way, like slave lag throttling, instrumentation, progress, etc.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Not
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Superseded
Series goal:
None
Implementation:
Deferred
Milestone target:
None
Completed by
Daniel Nichter

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.