Extracting HN Book Recommendations with the ChatGPT API, Vol. II

This is a follow-on from the original blog post Extracting HN Book Recommendations with the ChatGPT API.

Method used

In the original post I didn’t describe the process used to arrive at the results. I originally had a much longer post with much discussion about the process, but it was too long and the structure was poor, so I rewrote it in a more concise form. In the process I excluded the explanatory stuff. So in brief, here’s the process:

• Download all data from Hacker News (HN) api
• Load hn data into a duckdb database
• Query stories matching “book” as an individual word. Don’t include stories with a URL, as they are not usually Ask HNs.
• Get comments for those stories
• Use chatGPT chat API to extract book from comments
• Load the results into a Pandas dataframe to clean titles, remove subtitle (after colon), remove punctuation, lowercase, remove leading ‘the’.
• Load the book data into duckdb, and use sql to determine the top book titles.

OpenAI ChatGPT API timeouts

When using the chatGPT API, you definitely want to deal with timeout and not ready errors. If you call the api multiple times, there is a good chance there will be a timeout. (For my sentiment job below, there were 2 timeouts when calling the API 220 times.) You should also set the request_timeout, because the default seems to be 600, which means your program will be waiting around for 10 minutes if openai decides not to respond to your request. Here’s the code I used to deal with timeouts and to do retries:

Class Sentiment:

  def get_completion(self, messages, retry_time=15):
    model = "gpt-3.5-turbo"
    temp = 0
    try:
        completion = openai.ChatCompletion.create(model=model, messages=messages, temperature=temp, request_timeout=120)
        return completion
    except Exception as e:
        retry_time = e.retry_after if hasattr(e, 'retry_after') else retry_time * 2
        if retry_time > 60:
            retry_time = 60
        print(f"Exception {str(e)}. Retrying in {retry_time} seconds...")
        sys.stdout.flush()
        time.sleep(retry_time)
        return self.get_completion(messages, retry_time)

Sentiment

There was some discussion of the presence of Ayn Rand’s books on the top 50. Look, there are people who like them, and I think they appeal to a significant (enthusiastic?) minority of the HN crowd, possibly for the following reasons:

• people of an engineering mindset identify with the idea of building something great themselves.
• the American belief in individualism
• when you’re young, new ideas, books, movies, music, etc can be really mind-blowing and that generates enthusiasm. It might only be later that one’s education becomes more rounded in other areas and you might reconsider the message. Here’s a relevant XKCD

There’s always people who are enthusiastic about niche things, it’s not necessarily controversial that any community contains enthusiasts for something as relatively harmless as those books.

Someone mentioned that the list doesn’t account for people saying BAD things about a book, it will still get counted towards its ranking. I thought it might be an idea to see what the numbers are. I have used the chatGPT API to identify the sentiment of comments that recommend “Atlas Shrugged”, just to see how many mentions of it are negative. Note that we don’t need a model as powerful as Chatgpt 3 to determine sentiment. Any Machine Learning learning course will take you through using classification to determine sentiment, which will run fine on any laptop. However throwing together a few calls to chatGPT is really convenient, (much more so than digging through my old jupyter notebooks). It cost 6c to run the 220 API calls (plus 2 retries).

Here’s the prompt I used, where {book} is the book’s title and comment is the text of the individual comment from hacker news:

prompt = [
  {"role": "system", "content": f"Identify the sentiment of the following comment about the book {book}. Provide your answer as a number, where positive=1, neutral=0 and negative=-1. Answer in csv format, 1 column, no header"},
  {"role": "user", "content": comment},
]

So for “Atlas Shrugged”, the sentiment counts turn out to be:

So 14.5% of the comments were classed as negative by ChatGPT 3.5. 184 comments were positive or neutral. If we took that number, Atlas Shrugged would still be the 13th most-recommended book in my data.

Here’s the comments where the API didn’t return a valid number for sentiment, it looks like chatgpt returns multiple sentiment numbers if the comment contains a list. I used 0 for sentiment if the api didn’t return a valid number:

Top 51-100

Here are the top 51 to 100 ranked recommended books from my data: Again, I have thrown in some Amazon affiliate links to the books. Maybe I can earn a few measly dollars towards domain name subscriptions and cloud GPU usage.