Okay, @xukai92 @aartij22, I am fairly confident now that it's the functional test test_ctx_size() that's the problem.

1. In this PR, I'm starting us off with a simple, brute force doc splitting algorithm. This should be good enough to get us started, we can assess if we need something smarter with libraries like langchain after we've used this algorithm for a while. Essentially, the algorithm in this PR:

   • Takes a list of documents
   • Splits each documents in a way that the result will have documents that have at most 2400 words in them (based off of this math)

2. See this test PR, where I only introduce split_knowledge_docs for ilab generate, and the tests are failing at the same spot, with the same error.

• Interestingly enough, tests are failing at the EXACT same place in these PRs too:
  • Use the taxonomy schema to validate taxonomy yaml files #776
  • test #778
  • and of course, @aartij22 's initial PR add langchain for chunking #772

@aartij22 with the change to a brute force algorithm for splitting, we're eliminating the possibilities of anything risks being introduced by the introduction of langchain. Once
a) it becomes apparent that we do need a smarter way to split
b) we've written some tests around these changes to give us the confidence to iterate and become smarter

we should bring langchain PR back up coz it looks very promising, and possibly much more efficient than us introducing our logic (if we can lean on the community to do the heavy lifting for us, that's less maintenance for us 🎉 . ie if we have to iterate on the algorithm I introduced, that's a higher maintainance burden, so langchain is definitely on the cards)

@xukai92 I am fairly confident on these changes now, at least on the fact that it does not break any existing functionalities. I don't think we can justify investing any more time on looking at the functional tests. At least I don't think this PR should be blocked because of that. We can look at the functional tests as a follow up.

Okay, @xukai92 @aartij22, I am fairly confident now that it's the functional test test_ctx_size() that's the problem.

1. In this PR, I'm starting us off with a simple, brute force doc splitting algorithm. This should be good enough to get us started, we can assess if we need something smarter with libraries like langchain after we've used this algorithm for a while. Essentially, the algorithm in this PR:

   • Takes a list of documents
   • Splits each documents in a way that the result will have documents that have at most 2400 words in them (based off of this math)

2. See this test PR, where I only introduce split_knowledge_docs for ilab generate, and the tests are failing at the same spot, with the same error.

• Interestingly enough, tests are failing at the EXACT same place in these PRs too:

  • Use the taxonomy schema to validate taxonomy yaml files #776
  • test #778
  • and of course, @aartij22 's initial PR add langchain for chunking #772

@aartij22 with the change to a brute force algorithm for splitting, we're eliminating the possibilities of anything risks being introduced by the introduction of langchain. Once a) it becomes apparent that we do need a smarter way to split b) we've written some tests around these changes to give us the confidence to iterate and become smarter

we should bring langchain PR back up coz it looks very promising, and possibly much more efficient than us introducing our logic (if we can lean on the community to do the heavy lifting for us, that's less maintenance for us 🎉 . ie if we have to iterate on the algorithm I introduced, that's a higher maintainance burden, so langchain is definitely on the cards)

@xukai92 I am fairly confident on these changes now, at least on the fact that it does not break any existing functionalities. I don't think we can justify investing any more time on looking at the functional tests. At least I don't think this PR should be blocked because of that. We can look at the functional tests as a follow up.

Turns out, it was #788 all along! Closing in favor of #772