Instead of rolling a d20 for any check, players roll d6s. The number of d6s they roll is up to the player. However, for every roll of 1 on any die, the player loses 1d4 HP.
For 5e, rather than having skill bonuses, players would get either a certain number of riskless dice, which incur no penalty on rolling a 1, or the dice size increases to a d8. Proficiency gives riskless dice equal to their proficiency bonus and advantage gives a d8.
Players can roll dice one at a time, building up to the result they want, potentially taking HP loss as they go. Or they can roll a bunch of dice at once. If a player rolls multiple dice together, they get a +1 bonus to the roll for each die. They do not get this bonus unless rolling 2 or more dice at once. So rolling 3d6 would grant a +3 bonus, but rolling 1d6 does not grant a bonus.
After each die or group of dice is rolled, the player can declare they are done rolling, at which point the result is applied, then the HP loss is incurred. This allows players to push themselves to exhaustion (0 hp) in order to complete a check.
The idea behind this mechanic is to give players a little more control over their dice rolls with an added push your luck mechanic. Plus, it means players will never miss unless they choose to, which will reduce that frustration of missing an important attack or check. However, in order to do so, they will have to give up a valuable resource, their hit points.
This mechanic represents the player's ability to put a certain amount of effort into any one particular check. Swatting at a goblin is low effort and requires little risk to their HP, but could tire them out a little. But being able to wrestle a dragon is technically possible, but the player might pass out from exhaustion before they are able to actually do the deed. This also adds exciting gameplay moments of a player low on HP but pushing themselves to the brink of exhaustion in order to pass some critical check. Only to pass out after the task is complete.
It's up to the DM to determine which skills and situations this would apply to. For example, it is unlikely a character would be able to press themselves to passing out by thinking about History too hard.
Stamina
Every creature has HP but now also Stamina. There is no "max Stamina" technically. Stamina is limited by the creature's current HP, so as that goes down through damage, so does their Stamina.
Any exerting actions, attacking, magic, athletic skill, etc... uses Stamina. Any such action will have listed how much Stamina is required to do the action. Some actions have a base Stamina cost, such as attacking with a heavy weapon could cost 5 Stamina, but attacking with a light weapon could cost 2.
Some actions allow rolling Stamina costs, with the player being able to exert themselves to perform better. For example, lifting a heavy rock or attempting to magically break a curse. For these actions, players can roll as many times as they'd like, up to their class level, spending the Stamina for each die rolled.
Any creature may take the Rest action to restore Stamina equal to their Constitution score. A short rest restores all Stamina.
If a creature's Stamina reaches 0, they pass out from exhaustion.