Earlier this week, LinkedIn announced the open sourcing of Smart Argument Suite, a new Python tool designed to help users pass arguments through the command line interface and consume them in a “human-friendly” way. According to the company, while plenty of open-source projects offer CLI argument parsing, they don’t deal with the producer side. They were passing the arguments through.
The CLI becomes a producer and consumer problem: on the workflow generation side, you need to produce a set of arguments that are passed to the CLI to launch the jobs; on the other side, the established jobs would consume the arguments passed from the CLI,” Jun Jia and Alice Wu, senior staff software engineers at LinkedIn, wrote in a post.
The principles of the Smart Argument Suite are:
- It should be simple. The suite makes it easy to define an argument container object and pass it through a function call.
- It should be safe. The tool has a verifiable and testable systematic process with certain safety guarantees.
- It should be human-friendly. According to the team, the tool should be easy for humans to inspect or debug on the serialized form.
- It should be extensible. Users should be able to extend the support to the argument container classes if desired.
It’s a common scenario that an AI solution involves composing different jobs, such as data processing and model training or evaluation, into workflows and then submitting them to an orchestration engine for execution. There may be hundreds of thousands of such commissions per day at large companies such as LinkedIn, proposed and executed by multiple teams and engineers.
Jia and Wu wrote. “Any improvements in the tools used by machine learning engineers lead to significant improvements in productivity, which highlights the need for robust productivity infrastructure to support machine learning engineers.