Interesting But Problematic WM_MOUSEMOVE Issues

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
Hello there,

Long time ago i could not find a drawing program that i liked for Windows so i created one that could do things that i wanted a drawing program to do and be able to update it anytime with new features that would aide in drawing various things including schematics but also fine art. Recently however i noticed a problem that for some reason i never noticed before or didnt pay enough attention to in order to figure out that there was something actually very strange about drawing with the mouse.

Many drawing problems have a Zoom capability, and mine is no different. You can zoom in, you can zoom out. There is an interesting difference though in the way these two are related to the mouse movement.
When you zoom *IN* 2x (and this would be with any drawing program) the image gets 2 times larger and so when you move the mouse say one pixel to the right your graphic pixel position is really only 1/2 pixel to the right in the actual bitmap image. This doesnt seem to present any problem. However, when you zoom *OUT* and move the mouse one pixel to the right the distance the mouse pointer travels relative to the image is now 2 pixels. This creates a bit of a problem because if you draw one pixel for every mouse pixel movement, the line created by the mouse then appears as a dotted line with every other pixel remaining whatever color it was before the mouse was moved. So a black pixel color on a white background looks like a dotted black line with dots spaced 1 pixel apart. That is a problem because we dont want a dotted line when we want to draw a solid line. What is even worse is if we zoom out another time (like to 4x out) then the dots are even farther apart because now one mouse pixel movement spans 4 pixels on the actual bitmap image. I think i never noticed this before because i usually draw when the image is zoomed in, and that means that if i zoom in by 2x one pixel mouse movement covers just 1/2 pixel and so that means that each pixel gets drawn twice, which isnt really too much of a problem except with blend operations where the blend will occur twice. I am not so worried about that though because that's easy to fix by adjusting the blend ratio.

To make matters worse, WM_MOUSEMOVE only reports every 2 pixels but that is probably a separate issue.

So how to handle this. In the image attached i have drawn four circles. The four circles are drawn in a commercial program known as MSPaint that comes with most versions of the Windows operating system. I wanted to see how they handled this, even though i know they dont do the best with drawing programs.
The four difference circles are drawn with the following zoom levels:
1: zoom in 200 percent.
2: zoom standard, 100 percent (no zooming).
3: zoom out 200 percent.
4: zoom out 400 percent.
With these four different zoom levels i draw a circle for each one as close as possible to a free hand circle. The fact that they are not exact circles have nothing to do with this issue so it does not matter that they are not perfect circles. What is interesting though is that we can see that from 1 to 4 the circles get increasingly jagged, and i can guess why.
My guess is that when they get a WM_MOUSEMOVE message they detect current zoom level and the slope of the mouse movement (one of four possible values) and generate more pixel draw operations accordingly. For example, if the user moves the mouse one pixel to the right they fill in the second pixel to the right and also one pixel to the right if the zoom level is such that one pixel is skipped for that zoom level (that would be 100 percent in the drawing with teh circles as strange as that sounds, because the message only reports every 2nd pixel mouse movement as stupid as that is). So as the zoom is changed to 400 percent out, they have to fill in an additional 3 pixels, and the only way to get those locations from one mouse move report is to assume that they all have the same slope. That means either +1, -1, or 0 or completely vertical, and that is with any zoom level because there are no other mouse reports to use in the calculation. Perhaps we can calculate a better response using three mouse reports if there are that many that is. This may or may not work i havent thought about it much yet.

So i think that describes the issues associated with drawing with the mouse, at least with one pixel wide draw operations. OF course 2 pixel wide (2x2 pixels) doesnt have a problem when zooming in 2x but perhaps zooming in 4x will. Also, drawing complete lines is not a problem because there we naturally fill in pixels from one coordinate pair to another such as (1,1) to (5,5) or even (1,2) to (9,4); those pixel fills are all calculated pixel by pixel regardless of the zoom level.

So any ideas or comments would be interesting i think. Remember this issue is mainly only a problem when drawing with a single pixel wide line using the mouse where the mouse is moved across (or up and down) the screen where we would expect every pixel scanned by the pointer would be filled in. Also note that moving the mouse too fast skips pixels also, but that is not an issue. That occurs because Windows does not send more WM_MOUSEMOVE messages to the window than it can accept in a given time, but i have no problem with that.
 

Attachments

bogosort

Joined Sep 24, 2011
696
I've never worked on a drawing app, so I can't speak to the practicality of this, but it might be helpful to think of the bitmap as a linear space. Specifically, the app should be aware of two spaces: the bitmap itself (i.e., the values that would be written to a file) and the viewing space, which may or may not be equal to the bitmap space. From this perspective, zooming in/out is a linear transformation from the bitmap space to a new viewing space.

Your app is performing the drawing action on the viewing space, which doesn't map 1:1 with the bitmap space. Instead, the drawing actions should be performed on the bitmap space -- but to have the correct relationships preserved, the inverse view transformation should be applied to the drawing action before being applied to the bitmap space.

For example, a 2x zoom out is a scaling transformation by 1/2, which can be represented by some matrix Z that takes vectors in the bitmap space to vectors in the scaled viewing space. When the user clicks a pixel to draw on, that pixel is a vector in the viewing space and needs to be mapped back to the bitmap space. So, before executing the action, apply the inverse transform Z^-1 to the vector and draw on *that* pixel.
 

MrChips

Joined Oct 2, 2009
30,712
I agree.
You need to use a virtual graphics space of unspecified units. All graphics movement and drawing is performed on the virtual space. Then you transform the graphics space to bitmap space. By doing so your app can adapt to any pixel size and screen dimensions.
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
I agree.
You need to use a virtual graphics space of unspecified units. All graphics movement and drawing is performed on the virtual space. Then you transform the graphics space to bitmap space. By doing so your app can adapt to any pixel size and screen dimensions.
Thanks for the reply.

Well maybe you can elaborate a little. Are you maybe talking about the Windows API world transforms or something like that? I'd have to read up on that stuff again but maybe that would help. I never had to use those before because everything i did worked with logical pixels that were the same as real life pixels. This is totally different to me now but very interesting.
'
What i was thinking was that if i move the mouse one pixel when zoomed out 2x, that means the mouse had spanned 2 pixels on the bitmap anyway, so if i move it to the right it makes sense that it should fill in two pixels. But moving on a diagonal if i let it fill in two pixels per one pixel movement, i'll have to make sure it detects both x and y increments which from the graphics it looks like MSPaint does not do. With both x and y being accounted for maybe a diagonal line would look more like a one pixel wide line rather than a staircase like those circles in the image i posted.

This is all still a little new to me so really after years of having this program and updating it now and then i am just starting to think about it. I guess i either never realized it before or i did and just ignored it for the time being and went on to something else as i dont work on the program every day and sometimes weeks go by and i dont work on it. It would be cool to figure out what to do ehre though.

I still cant understand why WM_MOUSEMOVE does not report every pixel change though just every other one. For example if it reports x as 102 and y as 203 for one location, the next x to the right is 104 and the next y down is 205 when it should be x=103 and y=204. If x reports as 103 the next x will be 105, and if y reports as 204 the next y is 206, always either even or odd but never both. That's the strangest thing i could imagine they would want to do. I have to wonder if it has something to do with the mouse driver. I tried another mouse and it does the same thing but it might be using the same mouse driver or something. Just not sure what causes this yet and cant find anything on the internet that talks about it.
 

BobTPH

Joined Jun 5, 2013
8,813
I agree.
You need to use a virtual graphics space of unspecified units. All graphics movement and drawing is performed on the virtual space. Then you transform the graphics space to bitmap space. By doing so your app can adapt to any pixel size and screen dimensions.
Exactly. My CAD drawing program represents all coordinates as double precision floating point numbers representing inches. The zoom level is then nothing but a constant that translates those numbers to screen pixels. There is no problem with drawing lines at any scale, the mouse move gives you a distance in pixels, this is multiplied by a constant to get the move in inches. Windows even has this kind of transformation built in, but I don't use it because it is actually simpler to do the scaling yourself.

Bob
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
Exactly. My CAD drawing program represents all coordinates as double precision floating point numbers representing inches. The zoom level is then nothing but a constant that translates those numbers to screen pixels. There is no problem with drawing lines at any scale, the mouse move gives you a distance in pixels, this is multiplied by a constant to get the move in inches. Windows even has this kind of transformation built in, but I don't use it because it is actually simpler to do the scaling yourself.

Bob
Hi,

Thanks for the reply.

I am not certain that this is the same issue. I dont seem to have any problem scaling but perhaps more info would be good so i could understand other points of view.

See right now when i zoom out 2x and move the mouse one screen pixel to the right it might go from graphic x=0 to x=2 instead of from x=0 to x=1 which seems like i would want, but then in order for me to do that the mouse pointer would have to go from screen pixel x=0 to x=1/2, which not only can not be represented on the screen normally it would make the mouse appear to go 1/2 as fast across the screen as it did when not zoomed.

Just to be clear, not zoomed is 100 percent and then each graphic pixel equals one screen pixel, and zooming in 2x is 200 percent then one graphic pixel equals two screen pixels, and zooming out 2x is 50 percent and one graphic pixel equals 1/2 screen pixel.
Zooming out is the problem area. For example, it seems impossible to select some positions such as (1,1) because the mouse spans from x=0 to x=2 and from y=0 to y-2, so there is no way to position the mouse at (1,1) because it will not be possible to display that in zoom 50 percent.
Drawing lines seems like no problem, but now i see that could be an issue too. For example if not zoomed i can draw a line from (1,1) to (5,5) or from (1,1) to (6,6) no problem, but zoomed out to 50 percent i can never set the mouse to the point (1,1) nor (5,5) although i can at (6,6). So maybe this is just a consequence of working within the digital world.
As i said though i am just starting to think about this issue so i may be missing something simple.

Also note that when i talk about this right now i am also assuming that the mouse messaging system can report EVERY pixel move, when in reality it only reports every 2nd move, so the issues i talk about for zoomed 50 percent actually occur when zoomed 100 percent (no zoom) because the software can not detect a 1 pixel movement. That i am holding off on for now though as a separate issue so ignore this for now.
 
Last edited:

MrChips

Joined Oct 2, 2009
30,712
I may not have done this before hence I am just working this out in my head.

If you zoom out and then return back to 100% normal while working on the pixel data you will lose information (resolution). Hence you want to not zoom on the virtual space and zoom in pixel space only.

Since only objects are zoomed and not the cursor, the cursor has to be drawn in pixel space only. Doing so would not affect the size and position of the cursor (mouse pointer) when zooming.

----------------------

On another note, if you need to scale objects in virtual space then you need to preserve the precision. I would avoid using floating point since it requires too much CPU resources to compute. Fixed point representation would be my preferred method. Tagging on an additional byte to the values should give you sufficient scope for scaling up to x256. Three bytes (24 bits) would give you a range from 0 to 65535 with 1/256 resolution.

For example, scaling down 65 by a factor of 2 would give you 32.5

65 = 0000 0000 0010 0001 . 0000 0000
32.5 = 0000 0000 0001 0000 . 1000 0000

(The decimal point is implied and does not exist in coding.)

If 24-bit representation is too much and not required you can resort to two bytes, 12-bit integer and 4-bit fraction.
 

BobTPH

Joined Jun 5, 2013
8,813
Are you expecting to get a mouse move message for each pixel moved? Because it does not work that way.

I suspect you have a bug in your code that translates the coordinates.

Bob
 

BobTPH

Joined Jun 5, 2013
8,813
On another note, if you need to scale objects in virtual space then you need to preserve the precision. I would avoid using floating point since it requires too much CPU resources to compute
Todays CPUs have plenty of speed to use double precision floating point. My PCB CAD program is constantly updating the netlist, which requires looking for all connections between traces and pads while drawing and there is no noticeable delay.

Bob
 

MrChips

Joined Oct 2, 2009
30,712
I think the first AutoCAD program used integer arithmetic. When they switched to floating point performance took a big hit.

I am not an advocate for code bloat and demanding more memory space and CPU speed at every next generation SW and HW.

BTW - Why do Autodesk Inventor and Fusion 360 take so long to load and shutdown?
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
I may not have done this before hence I am just working this out in my head.

If you zoom out and then return back to 100% normal while working on the pixel data you will lose information (resolution). Hence you want to not zoom on the virtual space and zoom in pixel space only.

Since only objects are zoomed and not the cursor, the cursor has to be drawn in pixel space only. Doing so would not affect the size and position of the cursor (mouse pointer) when zooming.

----------------------

On another note, if you need to scale objects in virtual space then you need to preserve the precision. I would avoid using floating point since it requires too much CPU resources to compute. Fixed point representation would be my preferred method. Tagging on an additional byte to the values should give you sufficient scope for scaling up to x256. Three bytes (24 bits) would give you a range from 0 to 65535 with 1/256 resolution.

For example, scaling down 65 by a factor of 2 would give you 32.5

65 = 0000 0000 0010 0001 . 0000 0000
32.5 = 0000 0000 0001 0000 . 1000 0000

(The decimal point is implied and does not exist in coding.)

If 24-bit representation is too much and not required you can resort to two bytes, 12-bit integer and 4-bit fraction.
Hi,

I would have no problem with going to fixed point as i have used 64 bit integers in the past and it definitely speeds things up, mostly because the 8 core CPU i use only has 4 floating point units, a trick introduced by AMD and they got sued for it and i got some of my money back (not much though).

But unfortunately i still dont understand what you mean. When i zoom in 2x the graphic has to reduce by 1/2 along x and 1/2 along y, so the area of the screen covered is 1/4 of what it was with no zoom.
There's no way around that because the graphic must look smaller so it can be dealt with as smaller. When zooming in 2x the image appears 2 times larger along x and 2 times larger along y, and that is necessary for detail drawing work. Zooming out by 2x is mostly for when the image is too large for the screen and i want to view the entire image on one screen, otherwise it would go over the screen edges.
So anyway, when i zoom out by 2x and the image gets smaller it doesnt seem to matter how i store the pixels, float, 32 bit integer, etc. Right now i store them as 24 bit integers so 8 bits blue 8 bits green 8 bits red, standard BGR 24 bit color. These values are stored in a Windows bitmap structure. When something is changed (a pixel that is) the address is retrieved (actually several at once) and that address is used to store three 8 bit values that make up the 24 bit pixel color.
So i dont see how changing the resolution of the color bits for each pixel will change anything. It seems like the real problem is the mouse resolution isnt that good.

Now let's say i make a mouse move of two screen pixels equal one graphic pixel. That would mean the mouse would move 2 pixels for each image drawn pixel, and that would mean the mouse pointer would move along x at say position x=0 to position x=23 while the graphic image would only draw at pixels x=0 to x=11, which means the mouse would be far to the right while we see pixels far to the left being drawn. So the mouse pointer would no longer be directly over the pixel being drawn, and although that could be done i think it would be strange to have to draw a picture like that. I could make an illustration if you like so we can visualize this better.
But then again i am not sure i understand what you are suggesting yet. Maybe you could explain a little more or draw a quick graphic image or something to illustrate.
Thanks for the ideas too.
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
Are you expecting to get a mouse move message for each pixel moved? Because it does not work that way.

I suspect you have a bug in your code that translates the coordinates.

Bob
Hi,

What do you mean "it does not work that way". Do you know more about this issue?
If so i would like to hear more about it.
 

MrChips

Joined Oct 2, 2009
30,712
One may conceive of a basic implementation of computer graphics as simply as a matter of turning on and off pixels on a graphics screen. In order to add color information, each pixel is assign depth or value which can be as simple as an 8-bit value or a 24-bit value to produce 16 million colors and intensities. This is at the basic hardware level from the perspective of a graphics display.

This model works for some applications but one quickly discovers that it has its limitations. For example, how does one implement computer animation, transformations, dithering, anti-aliasing, blending, textures, 3-D graphics, CAD modelling, etc.? Added to that, how does one accommodate different hardware, screen size, aspect ratio, input devices, etc? One cannot find solutions to these challenges at the pixel level of the graphics display. One needs a different approach.

We toss out the connect the dots (or pixel) model. Instead we adopt a hierarchical model much like object-oriented programming. We describe all objects using properties including points, lines, surfaces, volumes, color, (in the case of 3-D objects. The model is simpler for 2-D objects since there is one fewer dimension.

As a simple example, a rectangle in 2-D has width, height, and position, not x1,y1 and x2,y2, or four corners. Units are not in pixels. Units are arbitrary values that are extensible using floating point or fixed point representation. By doing so the object can be rendered on any type of graphics display of any size and aspect ratio by performing the appropriate transformation which would include scaling, translation, rotation.

In summary, don't think of computer graphics as drawing lines on an x-y grid. Create mathematical models of your objects first and then have them rendered on to pixels. You don't zoom the model nor the pixels. Zooming is done in the transformation from model to pixels.
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
One may conceive of a basic implementation of computer graphics as simply as a matter of turning on and off pixels on a graphics screen. In order to add color information, each pixel is assign depth or value which can be as simple as an 8-bit value or a 24-bit value to produce 16 million colors and intensities. This is at the basic hardware level from the perspective of a graphics display.

This model works for some applications but one quickly discovers that it has its limitations. For example, how does one implement computer animation, transformations, dithering, anti-aliasing, blending, textures, 3-D graphics, CAD modelling, etc.? Added to that, how does one accommodate different hardware, screen size, aspect ratio, input devices, etc? One cannot find solutions to these challenges at the pixel level of the graphics display. One needs a different approach.

We toss out the connect the dots (or pixel) model. Instead we adopt a hierarchical model much like object-oriented programming. We describe all objects using properties including points, lines, surfaces, volumes, color, (in the case of 3-D objects. The model is simpler for 2-D objects since there is one fewer dimension.

As a simple example, a rectangle in 2-D has width, height, and position, not x1,y1 and x2,y2, or four corners. Units are not in pixels. Units are arbitrary values that are extensible using floating point or fixed point representation. By doing so the object can be rendered on any type of graphics display of any size and aspect ratio by performing the appropriate transformation which would include scaling, translation, rotation.

In summary, don't think of computer graphics as drawing lines on an x-y grid. Create mathematical models of your objects first and then have them rendered on to pixels. You don't zoom the model nor the pixels. Zooming is done in the transformation from model to pixels.
Hi,

Ok i think i know what you mean there when it comes to something like a rectangle or triangle or even a circle, but how do you mathematically describe something like a human face, or shaggy dog, or something like that?

But let's try to move from the very general to some detail then i think i can understand your view better.

In fact, i think we can even reduce this to 1 dimension by just assuming all we have to do is draw a horizontal line. So in this experiment, we have an original graphic that is 1000 pixels wide and 1 pixel high. It would have to be stored in a regular format file and to keep this simple let's say a bitmap where each pixel is represented by one byte and it is gray scale (0=black, 128=neutral gray, 255=white). Within the file the bytes would be predetermined from perhaps a download or someone else's drawing or a previous drawing that was rendered into a Windows 8 bit bitmap file (.bmp). We can ignore the header and just concentrate on the image bytes. The main point though is that they are predetermined and we are to render them into the drawing program memory space. Right now they are read in one at a time into successive bytes in memory, so when done, there are 1000 bytes and maybe some long or ulong variables that store the width and height. When the program opens, the mouse might be located near the beginning of the display on the image so that would be near the left side of the 'line' we got from the bitmap file. Let's also say that the image starts at the screen pixel position 0,0 so it's at the top left hand corner of the screen. That puts the mouse at position 0,0 and we can move it left and then later right (and for now we cant move it up or down because it's only one line of screen pixels, to keep this simpler).
Now if i move the mouse from the screen pixel on the farthest side on the left of the image to a screen pixel say 1 inch over to the right while keeping the horizontal constant, what would that do to the stored image in the manner you describe? If you wish to store the image in a different manner that's ok too but you would have to describe that also.
This is interesting, thanks.
 

BobTPH

Joined Jun 5, 2013
8,813
I am not an advocate for code bloat and demanding more memory space and CPU speed at every next generation SW and HW.
And I am not an advocate of adding complexity to make make a program run as fast as possible so the cpu is sitting in its idle loop 99% of the time.

My theory is that, given the platform you are targeting, use the simplest code that will give you good enough performance.

I actually started my PC CAD program using integers representing mils as the units. The code became way to difficult when doing things like intersections, so I switched to floating point simplifying the code. I saw absolutely no change in responsiveness.

I am tempted to do some benchmarks. I don’t think fixed point has any performance advantage over float on modern x86 processors.

Bob
 

BobTPH

Joined Jun 5, 2013
8,813
BTW - Why do Autodesk Inventor and Fusion 360 take so long to load and shutdown?
Do you have a solid state drive as your system disk? Recent programs have all been written by engineers who do, so they have no idea how slow their software is on a disk.

Fusion 360 used to take forever to load before I got a Terabyte SSD. Now it is at least tolerable.

Bob
 

BobTPH

Joined Jun 5, 2013
8,813
Hi,

What do you mean "it does not work that way". Do you know more about this issue?
If so i would like to hear more about it.
What I mean is a mouse move message can move by more than one pixel. It sounds as if you are drawing only one pixel for each mouse move.

Bob
 

MrChips

Joined Oct 2, 2009
30,712
You are still thinking about pixels and bitmaps.

Fonts and text rendering is the perfect example. Recall 5x7 bitmap fonts and dot-matrix printers?
Bitmap fonts are not easily scalable. Zoom and larger text showed lines as staircases or aliasing.

These are the reasons why Adobe PostScript and PDF were created. Vectors and mathematical models of curves are used to describe the shape of each object. These can then be scaled and then rendered to a screen of pixels.
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
You are still thinking about pixels and bitmaps.

Fonts and text rendering is the perfect example. Recall 5x7 bitmap fonts and dot-matrix printers?
Bitmap fonts are not easily scalable. Zoom and larger text showed lines as staircases or aliasing.

These are the reasons why Adobe PostScript and PDF were created. Vectors and mathematical models of curves are used to describe the shape of each object. These can then be scaled and then rendered to a screen of pixels.
Oh i guess you are talking about a pure math statement for each object or partial object. Not sure i want to get into that again but yes that would work probably to store the image. I did that with the alphabet long time ago before the age of the internet but it was easy for English letters, caps and lower case even.

But i dont think you yet described how you would render the graphic on screen when the mouse moves. It has to be in real time and the mouse pointer has to remain right over the part that is changed by the move.
 

Thread Starter

MrAl

Joined Jun 17, 2014
11,389
What I mean is a mouse move message can move by more than one pixel. It sounds as if you are drawing only one pixel for each mouse move.

Bob
Oh yes i have no problem with that. For example if i move the mouse very fast from point 1,1 to point 100,1 (along x) there will not be 100 reports, there will not even be 50 reports (keeping in mind i only get 1 report for every 2nd pixel during slow movement). That is not a problem i actually use that for special drawing effects. The unusual thing is that no matter how slow I move the mouse i only get 1 report for every 2 screen pixel movement of the mouse.
 
Top