2022-02-10 22:07:11 +00:00
|
|
|
# SPDX-License-Identifier: Apache-2.0
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
|
|
|
|
# ./blender.bin --background -noaudio --python tests/python/bl_pyapi_prop_array.py -- --verbose
|
|
|
|
import bpy
|
2021-07-29 00:52:11 +00:00
|
|
|
from bpy.props import (
|
|
|
|
BoolVectorProperty,
|
|
|
|
FloatVectorProperty,
|
|
|
|
IntVectorProperty,
|
|
|
|
)
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
import unittest
|
|
|
|
import numpy as np
|
|
|
|
|
2021-07-29 00:52:11 +00:00
|
|
|
id_inst = bpy.context.scene
|
|
|
|
id_type = bpy.types.Scene
|
|
|
|
|
|
|
|
|
|
|
|
# -----------------------------------------------------------------------------
|
|
|
|
# Utility Functions
|
|
|
|
|
|
|
|
def seq_items_xform(data, xform_fn):
|
|
|
|
"""
|
|
|
|
Recursively expand items using `xform_fn`.
|
|
|
|
"""
|
|
|
|
if hasattr(data, "__len__"):
|
|
|
|
return tuple(seq_items_xform(v, xform_fn) for v in data)
|
|
|
|
return xform_fn(data)
|
|
|
|
|
|
|
|
|
|
|
|
def seq_items_as_tuple(data):
|
|
|
|
"""
|
|
|
|
Return nested sequences as a nested tuple.
|
|
|
|
Useful when comparing different kinds of nested sequences.
|
|
|
|
"""
|
|
|
|
return seq_items_xform(data, lambda v: v)
|
|
|
|
|
|
|
|
|
|
|
|
def seq_items_as_dims(data):
|
|
|
|
"""
|
|
|
|
Nested length calculation, extracting the length from each sequence.
|
|
|
|
Where a 4x4 matrix returns ``(4, 4)`` for example.
|
|
|
|
"""
|
|
|
|
return ((len(data),) + seq_items_as_dims(data[0])) if hasattr(data, "__len__") else ()
|
|
|
|
|
|
|
|
|
|
|
|
# -----------------------------------------------------------------------------
|
|
|
|
# Tests
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
|
|
|
|
class TestPropArray(unittest.TestCase):
|
|
|
|
def setUp(self):
|
2021-07-29 00:52:11 +00:00
|
|
|
id_type.test_array_f = FloatVectorProperty(size=10)
|
|
|
|
id_type.test_array_i = IntVectorProperty(size=10)
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
scene = bpy.context.scene
|
|
|
|
self.array_f = scene.test_array_f
|
|
|
|
self.array_i = scene.test_array_i
|
|
|
|
|
2021-07-29 00:52:11 +00:00
|
|
|
def tearDown(self):
|
|
|
|
del id_type.test_array_f
|
|
|
|
del id_type.test_array_i
|
|
|
|
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
def test_foreach_getset_i(self):
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_set(range(5))
|
|
|
|
|
|
|
|
self.array_i.foreach_set(range(5, 15))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_set(np.arange(5, dtype=np.int32))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_set(np.arange(10, dtype=np.int64))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_get(np.arange(10, dtype=np.float32))
|
|
|
|
|
|
|
|
a = np.arange(10, dtype=np.int32)
|
|
|
|
self.array_i.foreach_set(a)
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_set(a[:5])
|
|
|
|
|
|
|
|
for v1, v2 in zip(a, self.array_i[:]):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
b = np.empty(10, dtype=np.int32)
|
|
|
|
self.array_i.foreach_get(b)
|
|
|
|
for v1, v2 in zip(a, b):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
b = [None] * 10
|
|
|
|
self.array_f.foreach_get(b)
|
|
|
|
for v1, v2 in zip(a, b):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
def test_foreach_getset_f(self):
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_i.foreach_set(range(5))
|
|
|
|
|
|
|
|
self.array_f.foreach_set(range(5, 15))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_f.foreach_set(np.arange(5, dtype=np.float32))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_f.foreach_set(np.arange(10, dtype=np.int32))
|
|
|
|
|
|
|
|
with self.assertRaises(TypeError):
|
|
|
|
self.array_f.foreach_get(np.arange(10, dtype=np.float64))
|
|
|
|
|
|
|
|
a = np.arange(10, dtype=np.float32)
|
|
|
|
self.array_f.foreach_set(a)
|
|
|
|
for v1, v2 in zip(a, self.array_f[:]):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
b = np.empty(10, dtype=np.float32)
|
|
|
|
self.array_f.foreach_get(b)
|
|
|
|
for v1, v2 in zip(a, b):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
b = [None] * 10
|
|
|
|
self.array_f.foreach_get(b)
|
|
|
|
for v1, v2 in zip(a, b):
|
|
|
|
self.assertEqual(v1, v2)
|
|
|
|
|
|
|
|
|
2021-07-29 00:52:11 +00:00
|
|
|
class TestPropArrayMultiDimensional(unittest.TestCase):
|
|
|
|
|
|
|
|
def setUp(self):
|
|
|
|
self._initial_dir = set(dir(id_type))
|
|
|
|
|
|
|
|
def tearDown(self):
|
|
|
|
for member in (set(dir(id_type)) - self._initial_dir):
|
|
|
|
delattr(id_type, member)
|
|
|
|
|
|
|
|
def test_defaults(self):
|
|
|
|
# The data is in int format, converted into float & bool to avoid duplication.
|
|
|
|
default_data = (
|
|
|
|
# 1D.
|
|
|
|
(1,),
|
|
|
|
(1, 2),
|
|
|
|
(1, 2, 3),
|
|
|
|
(1, 2, 3, 4),
|
|
|
|
# 2D.
|
|
|
|
((1,),),
|
|
|
|
((1,), (11,)),
|
|
|
|
((1, 2), (11, 22)),
|
|
|
|
((1, 2, 3), (11, 22, 33)),
|
|
|
|
((1, 2, 3, 4), (11, 22, 33, 44)),
|
|
|
|
# 3D.
|
|
|
|
(((1,),),),
|
|
|
|
((1,), (11,), (111,)),
|
|
|
|
((1, 2), (11, 22), (111, 222),),
|
|
|
|
((1, 2, 3), (11, 22, 33), (111, 222, 333)),
|
|
|
|
((1, 2, 3, 4), (11, 22, 33, 44), (111, 222, 333, 444)),
|
|
|
|
)
|
|
|
|
for data in default_data:
|
|
|
|
for (vector_prop_fn, xform_fn) in (
|
|
|
|
(BoolVectorProperty, lambda v: bool(v % 2)),
|
|
|
|
(FloatVectorProperty, lambda v: float(v)),
|
|
|
|
(IntVectorProperty, lambda v: v),
|
|
|
|
):
|
|
|
|
data_native = seq_items_xform(data, xform_fn)
|
|
|
|
size = seq_items_as_dims(data)
|
|
|
|
id_type.temp = vector_prop_fn(size=size, default=data_native)
|
|
|
|
data_as_tuple = seq_items_as_tuple(id_inst.temp)
|
|
|
|
self.assertEqual(data_as_tuple, data_native)
|
|
|
|
del id_type.temp
|
|
|
|
|
|
|
|
def test_matrix(self):
|
|
|
|
data = ((1, 2, 3, 4), (11, 22, 33, 44), (111, 222, 333, 444), (1111, 2222, 3333, 4444),)
|
|
|
|
data_native = seq_items_xform(data, lambda v: float(v))
|
|
|
|
id_type.temp = FloatVectorProperty(size=(4, 4), subtype='MATRIX', default=data_native)
|
|
|
|
data_as_tuple = seq_items_as_tuple(id_inst.temp)
|
|
|
|
self.assertEqual(data_as_tuple, data_native)
|
|
|
|
del id_type.temp
|
|
|
|
|
|
|
|
def test_matrix_with_callbacks(self):
|
|
|
|
# """
|
|
|
|
# Internally matrices have rows/columns swapped,
|
|
|
|
# This test ensures this is being done properly.
|
|
|
|
# """
|
|
|
|
data = ((1, 2, 3, 4), (11, 22, 33, 44), (111, 222, 333, 444), (1111, 2222, 3333, 4444),)
|
|
|
|
data_native = seq_items_xform(data, lambda v: float(v))
|
|
|
|
local_data = {"array": data}
|
|
|
|
|
|
|
|
def get_fn(id_arg):
|
|
|
|
return local_data["array"]
|
|
|
|
|
|
|
|
def set_fn(id_arg, value):
|
|
|
|
local_data["array"] = value
|
|
|
|
|
|
|
|
id_type.temp = FloatVectorProperty(size=(4, 4), subtype='MATRIX', get=get_fn, set=set_fn)
|
|
|
|
id_inst.temp = data_native
|
|
|
|
data_as_tuple = seq_items_as_tuple(id_inst.temp)
|
|
|
|
self.assertEqual(data_as_tuple, data_native)
|
|
|
|
del id_type.temp
|
|
|
|
|
|
|
|
|
Python: add foreach_get and foreach_set methods to pyrna_prop_array
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
2020-03-13 11:57:12 +00:00
|
|
|
if __name__ == '__main__':
|
|
|
|
import sys
|
|
|
|
sys.argv = [__file__] + (sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else [])
|
|
|
|
unittest.main()
|