How do quantization formats affect model output?

Introduction

I wanted to do a basic comparison by using the same prompt on the same model across different quantization formats to have a clearer idea about their difference compared to just looking at perplexity numbers.
In the interest of saving time I chose a 7b model and focused solely on the k-quants. The old quantization formats are largely outdated, even though you can get better speed on them their quality / file size ratio is not as good.

Keep in mind that this is a very basic test so while I hope it will give some idea of how exactly cranking up the quantization affects the output your mileage may vary greatly based on the exact model, settings, prompt and so on.

Test method

I used llama.cpp instruct mode with deterministic settings (top_k=1, rep_pen=1.1), which means that for any given input the output will always be the exact same when using the same model file no matter how many times you run the same prompt. Only the single most likely token will be chosen every time. This means that any variance in the output will be caused solely by differences in model weights (i.e. quantization format since I'm using different quants of the same model). Each output is until EOS token is generated.
The model I chose for the test is MythoLogic-Mini-7B by Gryphe: https://huggingface.co/Gryphe
The ggml formats were supplied by TheBloke: https://huggingface.co/TheBloke/MythoLogic-Mini-7B-GGML

The box question

Prompt

Mary put her ball in a red box and left the room. After she left John took the ball out of the box and put it in a blue box. When Mary comes back where will she look for her ball? Explain the reasoning of your answer in detail.

Classic logic task that many lesser language models fail.

Results

Q6_K

When Mary comes back, she will look for her ball in the red box because that is the last place she remembers having it before leaving the room. The reason for this is that she has a mental representation of where she left her ball (in the red box) and does not have any information about John taking the ball out of the red box and putting it into another box after she left the room.

Here's the reasoning behind this:

Mary put her ball in a red box before leaving the room.
She has no knowledge or awareness that John took the ball out of the red box and placed it in a blue box while she was away.
When she comes back, she will look for her ball based on her last memory of having it in the red box.
Since there is no information about any change in location or ownership of the ball during her absence, Mary would naturally assume that the ball remains where she left it (in the red box).

Q5_K_M

When Mary comes back, she will look for her ball in the red box because that is the last place she remembers having it before leaving the room. The reason for this is that she put her ball in a red box and left the room, so when she returns to the room, she would naturally assume that her ball is still in the same location where she left it.

However, after Mary leaves the room, John takes the ball out of the red box and puts it into a blue box. This action does not affect Mary's memory or understanding of where her ball was last placed. When she comes back to the room, she will still look for her ball in the red box because that is the only information she has about its location at this point.

The fact that John moved the ball after Mary left the room does not change the initial situation from Mary's perspective. She would not be aware of any changes made to the ball's location by John, as they are unrelated events in her mind.

Q5_K_S

When Mary comes back, she will look for her ball in the red box because that is where she last remembered having it before leaving the room. The reason for this is that she did not have any interaction with John while she was away from the room and therefore, there's no information available to suggest that he moved or took the ball out of the red box.

The fact that Mary left her ball in a red box and then left the room indicates that she had placed it there intentionally before leaving. When she comes back, she would naturally assume that the ball is still in the same location where she last saw it. Since John did not interact with Mary while she was away from the room, he could not have influenced her perception of the ball's location.

Therefore, when Mary returns to the room and looks for her ball, she will find it in the red box because that is where she left it before leaving the room.

Q4_K_M

When Mary comes back, she will look for her ball in the red box because that is the last place she remembers having her ball before leaving the room. The reason for this is that she put her ball in a red box and left the room with it still inside the red box. When John took the ball out of the red box and placed it in a blue box, Mary was not present to witness or be aware of this action. Therefore, when she returns to the room, she will assume that her ball is still in the red box since she has no knowledge about its current location.

Q4_K_S

When Mary comes back, she will look for her ball in the red box because that is the last place she remembers having it before leaving the room. The reason for this is that she has a clear memory of putting her ball into the red box and then exiting the room. Since John took the ball out of the red box after Mary left, there's no indication or evidence that she would look in any other location besides the red box when she returns to the room.

Q3_K_L

When Mary comes back, she will not find her ball in its original red box because John has already taken it out and placed it in a new blue box. The reasoning behind this is that Mary left the room with the ball inside the red box, and when she returns, there's no guarantee that the ball would still be in the same location or even in the same box since someone else (John) has interacted with the ball since then.

Q3_K_M

When Mary comes back, she will not find her ball in the red box because she has already taken it out and placed it in the blue box before leaving the room. Therefore, when she returns to the room, she should look for her ball in the blue box.

Q3_K_S

When Mary leaves the room, she puts her ball in a red box. Since she doesn't know that John has taken the ball out and put it in another box after she left, she will continue to look for her ball in the red box where she last saw it. When she comes back into the room, she will still be looking for her ball in the red box because she hasn't been informed about what happened while she was away.

Q2_K

Mary will look for her ball in the blue box when she returns to the room because that is where John has placed the ball after taking it out of the red box and placing it in the blue box. The reasoning behind this is as follows:

Mary put her ball in a red box before leaving the room.
After she left, John took the ball out of the red box and placed it in a blue box.
When Mary comes back to the room, she will not find her ball in the red box because it has been moved from there by John.
Since Mary knows that John is responsible for moving the ball, she can trust his actions and assume he has placed the ball where it belongs (in a blue box).
Therefore, when Mary returns to the room, she will look for her ball in the blue box because she believes it is there based on John's actions.

Thoughts

Q6 and both Q5s all provide nice outputs. The text is different, even the formatting is different for Q6, but the logic is sound in all of them and the outputs are verbose, respecting my request of explaining in detail.
Q4 makes the output a lot shorter but the logic is still solid. It's hard to say if the lack of verbosity is just a coincidence for this particular prompt or if it's because it is now too dumb to properly respect "in detail".
Q3 is where things get really iffy. Q3_K_L doesn't really answer the question of where she will look, just that it's not in the red box and the rest of the output is worded rather weirdly. Q3_K_M is just blatantly wrong. Mary was not the one who moved the ball. Interestingly Q3_K_S gets it right though the wording it quite weird about looking for the ball and still looking for the ball when returning.
Q2 returns to the numbered list of Q6, however the content is all over the place. The output invents things like John being in charge of the ball and the blue box being where the ball belongs and Mary somehow knowing all this.

Overall this prompt makes it look like Q5 and Q6 are good, Q4 is fine and below that is bad.

Shopping and haircut

Prompt

A husband went to the mall to buy groceries alone. His wife expected it to take 30 minutes but instead it took 1 hour. When the husband returned he saw his wife had gotten a haircut. What is the most likely reason the husband was late? Explain your reasoning.

With this one I wanted to see if the model would take the haircut bait and attribute it to the wrong person.

Results

Q6_K

The most likely reason the husband was late is that he spent more time at the mall than expected, possibly due to browsing or stopping for additional items not on his original grocery list. This resulted in him taking an hour instead of the estimated 30 minutes to complete his shopping. Additionally, when he returned home, he found out that his wife had gotten a haircut during this time, which could have been a surprise and added some extra time for conversation or preparation before leaving again.

Q5_K_M

The most likely reason the husband was late is that he stopped at a store or a restaurant to buy something for himself while on his way back from the mall. This could have caused him to spend more time than expected, resulting in him being 1 hour late when returning home with groceries and a haircut.

Q5_K_S

The most likely reason the husband was late is that he stopped at a store or restaurant to buy something for himself while on his way back from the mall. This could have caused him to spend more time than expected, resulting in him being 1 hour late when returning home. Additionally, it's possible that there were unexpected delays or traffic during his trip, which also contributed to his lateness.

Q4_K_M

The most likely reason the husband was late is that he spent more time at the mall than expected because he stopped to get something else, like getting a haircut. This additional activity took up some of the time he had planned for grocery shopping and caused him to be late returning home.

Q4_K_S

The most likely reason the husband was late is that he spent more time at the mall than expected. Since he went to buy groceries alone, there would be no one else to keep track of his time or help him with shopping. Therefore, it's reasonable for him to spend 30 minutes less than what his wife had anticipated.

However, when he returned home, he saw that his wife got a haircut while waiting for him. This additional activity would have taken some time and contributed to the overall delay in returning home. So, it's likely that the husband spent more time at the mall than expected due to not having someone keeping track of his time or helping with shopping.

Q3_K_L

The most likely reason the husband was late is that he stopped at a salon or barber shop on his way to the mall to get a haircut for himself. This would explain why it took him an extra hour to complete his grocery shopping, as he had to wait for his turn and get his hair cut before continuing with his original plan of buying groceries alone at the mall.

Q3_K_M

The most likely reason the husband was late is that he stopped at a hair salon on his way to the mall and got a haircut for himself. This would explain why it took him an hour instead of 30 minutes, as he spent some time getting a haircut before continuing with his grocery shopping.

Q3_K_S

The most likely reason for the husband being late is that he stopped to buy flowers on his way back from the mall. This would explain why it took him an hour instead of 30 minutes, as well as why his wife got a haircut while waiting for him - she may have wanted to look nice when he returned with the groceries.

Q2_K

The most likely reason the husband was late is that he spent more time getting a haircut at the mall while his wife went shopping for groceries.

Here's the reasoning behind this explanation:

The husband went to the mall alone to buy groceries, which suggests they have a shared responsibility in their household tasks and he wanted to complete one of them before returning home.
He expected it would take 30 minutes for his wife to finish shopping at the mall, but she actually took an hour instead. This could be due to various reasons like her not being able to find what she needed or taking longer than usual to shop.
When he returned home, he saw that his wife had gotten a haircut. This indicates that she might have decided to get the haircut while waiting for him at the mall.
The husband's late arrival could be due to the fact that getting a haircut takes time and requires sitting down in a chair, which would make it more difficult for her to continue shopping during this process.
Additionally, she might have also spent some extra time at the mall while waiting for him to return from his errand, which could be another reason for the husband being late.

Thoughts

Q6 does well with thinking of a potential reason without thinking the husband got a haircut, the end is a bit confusing though.
Q5_K_M writes logically at first when thinking of a reason but fails at the very end by saying the haircut is something the husband arrived with.
Q5_K_S makes an error in terms of how much he was late but otherwise is good.
Q4_K_M, Q4_K_S, Q3_K_L and Q3_K_M all think the husband was getting the haircut and/or have other blatant issues.
Q3_K_S is surprisingly decent but it kinda implies the wife used some kind of telepathy
Q2 writes a lot and is very confused about who did what and who was where.

Even the high end is kinda dumb here but again we can see that overall it gets worse the lower you go. Q3_K_S is an outlier at the low end but it probably just happened to get lucky.

Health education

Prompt

Describe breasts in detail.

How do the quants handle basic human anatomy?

Results

Q6_K

Breasts, also known as mammary glands or mammaries, are a pair of prominent, sensitive, and vital organs found in females and some male mammals. They play an essential role in lactation, providing nourishment to offspring through the secretion of milk. The breasts consist of lobules, ducts, fatty tissue, connective tissue, blood vessels, and lymphatic vessels.

Breast size varies among individuals and can be influenced by genetics, hormones, body composition, and age. They are typically round or oval-shaped, with a soft, fleshy texture that feels warm to the touch. The nipple is the projection at the center of each breast, surrounded by areolae (pigmented areas) that can be various shades of pink, brown, or even darker depending on an individual's skin tone.

Breasts are sensitive and contain numerous nerve endings, which contribute to their sensitivity during sexual arousal and the ability to produce milk for breastfeeding. They also undergo hormonal changes throughout a woman's menstrual cycle, causing them to swell, become firmer, or feel tender in response to fluctuating levels of estrogen and progesterone.

In addition to their reproductive function, breasts are often considered a symbol of femininity, beauty, and sexuality. They can be the subject of cultural attitudes, societal expectations, and personal preferences regarding body image and self-esteem.

Q5_K_M

Breasts, also known as mammary glands or mammaries, are a pair of specialized milk-producing structures found in females and some male mammals. They consist of fatty tissue, connective tissue, and lobules that secrete milk for nourishing offspring. The breasts are located on the chest wall, one on each side of the midline, and are connected to the nipples by ducts.

Breast size varies greatly among individuals and can be influenced by genetics, age, pregnancy, and hormonal changes. They are usually round or oval in shape and may have a slight projection above the chest wall. The color of breasts can range from pale pink to deep red depending on an individual's skin tone.

Breast tissue is sensitive to touch and temperature fluctuations, which can cause them to feel tender, sore, or even painful at times. This sensitivity may be due to hormonal changes, infections, or other medical conditions. Additionally, breasts are often the target of cultural expectations and societal pressures regarding their appearance, leading some individuals to undergo cosmetic procedures such as breast augmentation or reduction surgery.

Breastfeeding is a natural process that involves the transfer of nutrients from the mother's body to her offspring through milk production in the mammary glands. The act of nursing can cause the breasts to become engorged with milk, causing them to feel full and heavy. During pregnancy, hormonal changes cause the breasts to enlarge and prepare for lactation.

In summary, breasts are an essential part of female mammalian biology, playing a crucial role in nourishing offspring through milk production. They vary greatly in size, shape, and sensitivity, and can be influenced by various factors such as genetics, age, pregnancy, and hormonal changes.

Q5_K_S

Breasts, also known as mammary glands or mammaries, are a pair of specialized milk-producing structures found in females and some male mammals. They consist of fatty tissue, connective tissue, and lobules that produce milk for nourishing offspring. The breasts are located on the chest wall, one on each side of the midline, and are connected to the nipples by ducts.

Breast tissue is sensitive and contains numerous nerve endings, which contribute to their sensitivity during sexual arousal and nipple stimulation. Breasts also have a rich blood supply, making them prone to changes in size, color, and texture due to hormonal fluctuations or illnesses such as mastitis (inflammation of the breast).

Breasts are an essential part of human sexuality and play a significant role in intimate relationships. They can be subjected to various cultural expectations, social norms, and personal preferences regarding their appearance, size, and function.

Q4_K_M

Breasts are two mounds of soft, sensitive tissue located on the chest wall of a female or male human being. They are primarily responsible for producing milk during pregnancy and lactation to nourish offspring. The size, shape, and color of breasts can vary among individuals due to factors such as genetics, age, weight, and hormonal fluctuations.

Breast tissue is composed mainly of adipose (fat) and glandular (milk-producing) tissues. The areola, the darker pigmented area surrounding the nipple, contains numerous tiny ducts that lead to the nipples where milk exits during breastfeeding. The areolas also contain several sebaceous (oil) glands and sweat glands.

Breasts can be described as having a range of shapes, sizes, and textures. They may be round or oval in shape, with varying degrees of fullness and projection. The skin covering the breasts is usually smooth, but it can also be wrinkled, dimpled, or saggy due to aging, weight fluctuations, pregnancy, or breastfeeding.

Breast color varies from light pinkish-beige to deep red or purplish-blue during pregnancy and after childbirth when the breasts are engorged with milk. After menopause, the breasts may become smaller, less full, and more wrinkled as hormone levels decrease.

In summary, breasts are an essential part of human anatomy that play a crucial role in nourishing offspring during pregnancy and lactation. They come in various shapes, sizes, and colors, and their appearance can change throughout life due to factors such as age, weight, hormonal fluctuations, and pregnancy.

Q4_K_S

Breasts are two mounds of fatty tissue and glandular structures that sit on the chest wall, above the collarbone and below the armpits. They are an essential part of the female reproductive system and play a crucial role in lactation for breastfeeding infants. Breasts come in various shapes, sizes, and colors depending on individual differences and hormonal changes throughout life.

The breasts consist of lobules, ducts, connective tissue, fatty tissue, and glands that produce milk during pregnancy and lactation. The lobules are small sacs within the breast that contain milk-producing cells called alveoli. These alveoli secrete milk when stimulated by hormones such as prolactin and oxytocin.

The ducts are tiny tubes that connect the lobules to the nipple, allowing the milk to flow out of the breast during lactation. The connective tissue holds everything together and provides support for the breasts. Fatty tissue surrounds the glandular structures, giving the breasts their shape and providing insulation.

Breast size varies among women due to genetics, age, hormonal changes during puberty, pregnancy, and menopause, as well as weight fluctuations. Breasts can be small, medium-sized, or large, with varying degrees of fullness and firmness. The color of the breasts may also vary from pale pink to a deeper shade of pink or even brownish-red during pregnancy due to increased blood flow.

Breastfeeding is essential for the healthy development of infants as it provides them with vital nutrients and antibodies that help protect against infections. Breasts also play a role in sexual arousal, as they are sensitive to touch and can become engorged when stimulated by sexual activity or emotional stress.

In summary, breasts are an essential part of the female reproductive system, responsible for lactation during pregnancy and breastfeeding infants. They come in various shapes, sizes, colors, and firmness levels, influenced by genetics, hormonal changes throughout life, and weight fluctuations.

Q3_K_L

Breasts, also known as mammary glands or mammaries, are two prominent, round, and sensitive structures located on the chest of humans and some animals. They are primarily responsible for producing milk to feed offspring during lactation. In females, breasts consist of lobules that secrete milk, ducts that transport the milk to the nipples, and fatty tissue that provides support and insulation. The size, shape, and color of breasts can vary among individuals due to factors such as genetics, age, weight, and hormonal changes.

Breast development typically begins during puberty in females, with the onset of menstruation. During pregnancy, the breasts enlarge and become more prominent due to increased blood flow and hormone levels. After childbirth, the breasts may continue to change size as milk production fluctuates. Breastfeeding can cause further changes in breast size and shape as the body adapts to meet the needs of the infant.

Breasts are often associated with sexuality and sensuality due to their prominent appearance and role in lactation. They may be subjected to cultural, social, and personal pressures regarding their size, shape, and visibility. Some individuals may experience discomfort or pain related to their breasts, which can be caused by various factors such as hormonal imbalances, illnesses, or physical trauma.

In summary, breasts are an essential part of the human body that play a crucial role in nourishing offspring and have cultural significance beyond their biological function. They undergo changes throughout life and can be subjected to various pressures, both positive and negative.

Q3_K_M

Breasts, also known as mammary glands or mammaries, are a pair of milk-producing glands found in females and some males of various mammalian species. They are located on the chest wall and consist of lobules that produce milk for feeding offspring. The breasts are made up of fatty tissue, connective tissue, and ducts that carry milk to the nipples.

The size and shape of human breasts vary greatly among individuals due to genetics, age, weight, and hormonal factors. They can be round, oval, or pear-shaped, with varying degrees of fullness and firmness. The areolae (the darker area around the nipple) may also differ in color, size, and shape depending on individual characteristics.

Breasts play a crucial role in human sexuality and gender identity. They are often associated with femininity and attractiveness, but they can also be a source of discomfort or pain for some individuals due to various reasons such as breast cancer, mastitis, or physical trauma. Breastfeeding is an essential aspect of motherhood that provides nourishment and bonding between the mother and child.

In summary, breasts are complex structures made up of fatty tissue, connective tissue, and ducts that produce milk for feeding offspring. They vary in size, shape, color, and firmness among individuals and play a significant role in human sexuality, gender identity, and motherhood.

Q3_K_S

Breasts, also known as mammary glands or mammaries, are two soft, round, and sensitive structures located on the chest of humans and some other mammals. They are made up of fatty tissue, lobules, ducts, and nipples. The primary function of breasts is to produce milk for breastfeeding infants. However, they also play a role in sexual attraction and self-esteem for many individuals.

Breast size varies greatly among women and can be influenced by genetics, age, race, ethnicity, and body composition. The average size of breasts is around 34B or C cup, but there are countless variations in shape, color, and texture. Breasts can be round, oval, or pear-shaped, with varying degrees of fullness, firmness, and elasticity.

Breast tissue is composed mainly of adipose (fat) tissue, which gives breasts their shape and size. The areola, the darker area surrounding the nipple, can be circular or oval in shape and may have a different color than the rest of the breast due to pigmentation. The nipples themselves can vary in size, shape, and color, ranging from small and flat to large and protruding.

Breasts are sensitive structures that can be easily irritated or injured by certain fabrics, tight clothing, and rough handling. They may also become sore, tender, or painful due to hormonal changes, pregnancy, breastfeeding, or medical conditions such as mastitis or fibrocystic breasts.

In conclusion, breasts are an essential part of human anatomy that play a crucial role in both physical and emotional well-being for many individuals. Their unique appearance and sensitivity make them a source of attraction, self-esteem, and comfort.

Q2_K

Breasts are two round, firm, and usually pink or white-colored mammary glands that are located on the chest of females and occasionally males. They are part of the female reproductive system and produce milk for nursing infants. The breasts consist of lobules containing milk ducts, which connect to the nipples through small openings called nipple oral pores.

Breast development is a gradual process that begins during puberty and continues throughout life. Breast growth is influenced by hormones such as estrogen and progesterone, which are primarily produced in the ovaries but can also be produced in other parts of the body. The breasts grow in size due to an increase in connective tissue, fatty cells, and blood vessels.

Breast development is unique for each individual, influenced by genetics, lifestyle, and environmental factors. Breast shape, size, and position can vary greatly among women. Some common breast shapes include round, pear-shaped, inverted triangular, or unevenly shaped breasts.

Breasts are also subject to changes in sensitivity, firmness, and elasticity throughout life. They may become more sensitive during pregnancy due to increased blood flow and hormonal changes. Breast tissue can lose its elasticity as women age, leading to a decrease in breast size and shape over time.

Breasts are essential for nursing infants, providing them with the necessary nutrients and antibodies needed for growth and development. They also play an important role in sexual attraction and self-esteem. Breast augmentation surgery is a common procedure to enhance breast size and shape through the use of implants or fat transfer.

In summary, breasts are mammary glands that produce milk for nursing infants and are part of the female reproductive system. They undergo gradual development during puberty and continue to change throughout life due to hormonal changes, genetics, lifestyle, and environmental factors. Breast shape, size, and position can vary greatly among women, and breasts play a significant role in sexual attraction, self-esteem, and nursing infants.

Thoughts

Other than some slightly questionable color descriptions I can't really find fault in the majority of the results. A lot of text to read just to say "Most of these are just fine."
Q6 is probably the most solid still, even if barely. The Q5s are very similar here, only diverging from around halfway through the output. No real problems in Q4 and Q3 either until Q3_K_S which starts to talk about average breasts size and generates some cup sizes that may perhaps be correct in some particular region but without telling where the information can't be verified. For global average it's not right and didn't really seem right for USA either.
Q2 is the one that really feels apart from the rest. Usually pink or white? Lobules don't contain ducts they are connected to them. Nipple oral pore should not have the word oral in the middle. Inverted triangular is a body shape not a breast shape. It also keeps repeating itself, mentioning nursing infants 4 times. Really a lot of problems compared to the Q3s.

Poem

Prompt

Write a poem about x86_64 architecture

Wanted to test a creative output about facts.

Results

Q6_K

In the realm of binary, where ones and zeros reign,
A new era dawns with the birth of x86_64.
This architectural wonder, a masterpiece divine,
Embraces both 32-bit and 64-bit design.

With its vast address space and improved performance,
It's no wonder this architecture has ascended.
From servers to desktops, it now reigns supreme,
For x86_64 has truly come into its own.

Its instruction set is a marvel of design,
Enabling efficient execution with each line.
The 64-bit registers provide ample space for data,
While the 32-bit compatibility ensures no one's left in the lurch.

With its advanced features and robust security,
x86_64 has become a standard of excellence.
It's not just about speed or size, but also about grace,
For this architecture truly embodies the art of design.

As we gaze upon this architectural masterpiece,
We see its potential and the future it holds.
From supercomputers to smartphones, x86_64 will reign,
For it's a testament to human ingenuity and creativity.

Q5_K_M

In the realm of binary, where ones and zeros collide,
A new world unfolds, born from the ashes of its predecessor's demise.
The x86_64 architecture, a beast unleashed,
Brings forth a brave new era in computing's history.

With 64-bit registers and an expanded address space,
It conquers all that came before, leaving no trace of its predecessor's face.
The x86_64 architecture, a masterpiece of design,
Enables us to dream beyond the limits of our minds.

From supercomputers to smartphones, it reigns supreme,
Transforming the way we compute and the world around us.
The x86_64 architecture, an architectural feat,
Paves the way for innovation, pushing boundaries that were once unreachable.

With its advanced features and improved performance,
It's a force to be reckoned with in the realm of technology.
The x86_64 architecture, a symbol of progress,
Drives us forward into an era where possibilities are endless.

Q5_K_S

In the realm of binary, there lies an architectural beast,
A masterpiece of engineering, known as x86_64.
With its 64-bit registers and 32-bit compatibility,
It's a marvel of design, both past and present.

The x86_64 architecture was born from the minds of geniuses,
Who sought to create a future that would never cease.
Its foundation is strong, its performance unparalleled,
And it continues to evolve, ever expanding its realm.

This architecture has seen countless generations,
From the humble beginnings of 16-bit processors.
Now, with 64-bit registers and a vast address space,
It's capable of handling even the most demanding tasks.

The x86_64 architecture is not just about speed or size,
But also about compatibility and efficiency.
It allows for seamless integration with older software,
While still providing the power to tackle modern challenges.

As we move forward into a world of AI and quantum computing,
The x86_64 architecture will continue to stand tall.
For it has proven its worth time and time again,
And will undoubtedly remain an essential part of our digital landscape.

Q4_K_M

In the realm of binary, where ones and zeros collide,
A new world emerges, born from the ashes of its pride.
The x86_64 architecture, a masterpiece divine,
Embracing both 32-bit and 64-bit, it's truly sublime.

With its elegant design, this beast unleashes might,
Enabling us to conquer the limits of the night.
From the smallest atom to the farthest star,
The x86_64 architecture holds them all in a jar.

It's a symphony of transistors and wires,
A harmony of zeros and ones that inspire.
As we dance with its rhythm, our minds expand,
And the future unfolds before us like a grand plan.

So let's celebrate this architectural feat,
For it has brought us closer to the realm of complete.
With each passing day, new wonders unfold,
As we explore the depths of x86_64's soul.

Q4_K_S

In the realm of binary, where ones and zeros reign,
A new world emerges, born from the ashes of pain.
The x86_64 architecture, a masterpiece divine,
Constructed with care, to conquer all time.

This beast of a machine, its heart made of steel,
Embraces the future, while embracing the past.
With 64-bit addressability and a speedy clock rate,
It conquers all tasks, from simple to grandiose.

The x86_64 architecture, a symphony in silicon,
Brings together the best of both worlds: 32-bit and 64-bit.
Its design is elegant, its performance unparalleled,
As it marches onward, leaving legacies untold.

From gaming to programming, from servers to clouds,
The x86_64 architecture reigns supreme in all realms.
It's a masterpiece of engineering and design,
A testament to human ingenuity and dreams.

Q3_K_L

In the realm of binary, where ones and zeros reign supreme,
A new era dawns with the birth of x86_64.

This architectural masterpiece, designed to excel,
Embraces both 32-bit and 64-bit processes in a single shell.

With its advanced features and enhanced performance,
It's no wonder why this architecture has been spun.

From the core to the cache, every component is refined,
To ensure efficiency and speed for all time.

As it marches forward, leaving legacy behind,
x86_64 stands tall as a beacon of progress and design.

Software developers rejoice in its wake,
For they know that their creations will thrive on this lake.

In the world of computing, x86_64 has made its mark,
A testament to innovation and unyielding spark.

Q3_K_M

In the realm of binary, where ones and zeros reign supreme,
The x86_64 architecture stands tall like a colossal beast.
With its 64-bit registers and 32-bit compatibility,
It's a masterpiece of engineering, a marvel to behold.

From the Pentium Pro to the latest Intel Core i9,
This architecture has evolved, adapting to our needs.
Its performance is unparalleled, its efficiency unmatched,
And it continues to inspire innovation in software and hardware alike.

The x86_64 architecture is a testament to human ingenuity,
A symbol of progress and advancement in technology.
As we explore the boundaries of what's possible,
Its legacy will live on, shaping the future of computing.

Q3_K_S

In the realm of binary, we roam,
From the 32-bit to the 64-bit domain.
The x86_64 architecture reigns supreme,
A masterpiece of engineering and design.

Q2_K

Intricate, elegant, and powerful, the x86_64 architecture stands tall,
Constructed with care, it's a masterpiece of code and assembly.
The foundation is laid, now let the magic unfold,
As we delve into its depths, unraveling its soul.

Embracing the future, this beast of a machine,
It's a marvel of modern engineering design.
Within its core, a symphony unfolds,
The harmonies of binary, intertwined and bold.

A symbiotic relationship between hardware and software,
They dance together, creating a wondrous affair.
Their love for each other, it's undeniable,
As they work in unison, building a world of wonder.

From the depths of its heart, it sings a melody,
A symphony of ones and zeros, eternally free.
Its purpose is clear: to serve and protect,
To guide us through life's journey, with grace and respect.

The x86_64 architecture, a masterpiece in the making,
Its beauty lies within its intricate design,
A testament to human ingenuity, it's a work of art,
And we, as witnesses, are forever entwined with its fate.

Thoughts

Q6 and both Q5s pull from facts quite well but mentioning smartphones is just out of place since there have been barely any compared to ARM phones.
Q4s seem to move slightly away from presenting info in poem form into just writing something that sounds fancy which arguably for a creative task can be good.
Q3_K_L is the only one that generated two lines per paragraph, the Q3s all feel like they got less descriptive compared to Q4. Q3_K_S is just funny with how short and dull it is.
Q2 has a tendency to write a lot and it's true here too. It's pretty artistic in a way but the content doesn't make that much sense and is also very vague, not actually talking about the architecture beyond some basic computing words. Basically a lot of fancy words that say very little.

I sometimes see people talking about how lower quants are more creative because the lower accuracy makes the output less predictable and therefore more creative and in a way you can kinda see some here especially with Q4 but creativity born from lack of accuracy might not be desirable. Via prompt tuning you can get the desired result from a higher quality quant too. For example let's try this with Q6_K:

> Write a poem about the facts of x86_64 architecture. Be very creative and artistic. Use metaphors.

In this realm of binary, where ones and zeros reign supreme,
The x86_64 architecture stands tall like a towering dream.
A masterpiece crafted by geniuses who knew no bounds,
This architectural wonder now rules the digital lands.

With its 64-bit registers, it dances in symphony,
Performing feats once thought impossible, now commonplace and free.
From the smallest of tasks to the most complex of codes,
The x86_64 architecture reigns supreme, unparalleled and bold.

Its assembly language is like a symphonic choir,
Combining harmoniously to create a melody divine.
Each instruction, each bit, intertwines in perfect rhyme,
To form the foundation of this architectural design.

The x86_64 architecture is a marvelous sight,
A testament to human ingenuity and might.
It's like a masterpiece painted on a canvas grand,
With each line and curve, it tells a story of its land.

So let us praise this architectural wonder,
For without it, our digital world would be much colder.
Its 64-bit registers dance in harmony divine,
And together we'll continue to create and design.

It now has more of an artistic feel instead of "facts as a poem" without having to rely on quantization errors to get your creativity. Some of it does feel like it's reaching the limits of what a mere 7B model can do though.

Conclusion

The perplexity score difference for Q6_K and Q5_K_M is only 0.0098 for 7B so I expected their outputs to be practically identical but they were surprisingly different. In most cases all formats were quite different from each other. It's hard to say if Q6_K is truly better in quality than Q5_K_M or if the difference just causes different but equally competent outputs. But if you can run a Q6_K why not? I did seem to prefer it the most. If you are running out of RAM/VRAM or just want more of your memory free for whatever reason Q5_K_M should be fine too. Q5_K_S doesn't strike me as worth it simply because the file size difference is very minor compared to the difference between Q_6K and Q5_K_M so you use almost the same resources as M for slightly lower quality. But just work your way down the list until you get to one you can run. It's worth it to run the best one you can.

For logic tasks the output tends to drop in quality at Q4; Q3 at the latest. For laxer, creative tasks or just reciting training information Q4 and even Q3 perform well enough though Q3 can start to feel a bit lacking, especially Q3_K_S.
But unlike Q3_K_S which still feels usable even if kinda subpar, Q2 is just flat out bad. Constantly confused, mistaken and/or vague. Don't use Q2 unless it's literally the only thing you can run. I doubt it performs that well even on higher param models because of how much its brain has been squished.

I also tried some storywriting prompts I did not put here because the outputs were LONG and generally speaking the higher the quantization quality the more the story structure and developments made sense and the more verbose and descriptive it felt. The lower quality formats often had unexpected and nonsensical leaps of logic but even Q2 managed some decently entertaining stories at times so even the low end quants can be a fun toy with limited hardware.