popoto.transfer.import_¶
popoto.transfer.import_
¶
Import records from a JSON Lines export produced by :mod:popoto.transfer.
Keys are preserved by default. Records are saved one at a time -- deliberately
not through bulk_create -- because an external pipeline makes save()
return the pipeline for every record and destroys per-record observability,
which the reconciliation ledger depends on.
preserve_keys=False is the opt-out: every record gets a freshly minted key
and every field that declares a reference to another record is rewritten to
point at the new one. That mode reads the record stream twice (see
_spool_and_mint), because a reference on record 3 can point at record 4000
and the complete old-to-new map must exist before the first write. The default
path is untouched by it: one forward pass, no buffering, no map.
BATCH_SIZE = 500
module-attribute
¶
Records per conflict-check / reconciliation batch.
REMAPPED_REFERENCES_KEY = '__remapped_references__'
module-attribute
¶
In-memory slot on a record dict carrying its rewritten reference strings.
Set by :func:_remap_record and consumed by :func:_process_batch only when
that batch was produced by the regenerating path. Export never writes this
name, and the preserving path pops it and throws it away, so a file that
happens to carry one cannot use it to set relationship values.
import_records(model_class, stream, on_conflict='error', on_write_gate='reject', on_embedding_mismatch='error', preserve_keys=True, key_map=None, uow=None)
¶
Import records from a JSON Lines export into model_class.
By default keys are preserved, so re-running an import converges rather
than duplicating, and every Relationship value and application-level
key pointer keeps pointing at the same record.
preserve_keys=False regenerates every key instead, and rewrites the
references it can. Three properties of that mode matter before choosing
it:
- It is not idempotent. Every other mode converges on a re-run; this one mints fresh keys each time, so running it twice against the same destination leaves two copies of every record.
- The remap is partial, by design. Only fields that declare a
reference are rewritten --
Relationshipis the only one Popoto ships. An application-level pointer stored in a plainField(type=str)is indistinguishable from ordinary text, is never rewritten, and will dangle. Popoto does not guess: a heuristic that scanned strings for key-shaped values would turn a documented limitation into occasional silent corruption. The report names both counts so a dangling reference is visible in the run that created it. - It requires a mintable key. The destination model's key must be
exactly one
auto=Truefield; anything else is refused before a byte is written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_class
|
'type[Model]'
|
The destination Model class. |
required |
stream
|
TextIO
|
A text file-like object positioned at the manifest line. |
required |
on_conflict
|
str
|
What to do when the destination already holds a key.
|
'error'
|
on_write_gate
|
str
|
|
'reject'
|
on_embedding_mismatch
|
str
|
|
'error'
|
preserve_keys
|
bool
|
|
True
|
key_map
|
'dict[str, str] | None'
|
Only valid with |
None
|
uow
|
Any
|
A Postgres unit of work ( |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
An |
ImportReport
|
class: |
ImportReport
|
skipped, rejected, errored, or partial, with a reason per non-landed |
|
ImportReport
|
record. Under |
|
ImportReport
|
mapping this run actually wrote, merged over the seed -- mints whose |
|
ImportReport
|
record did not land are pruned; see :class: |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a policy argument is not one of its allowed values, or
if |
ModelException
|
If the manifest is missing, the format version is
unsupported, the model name does not match, an embedding
provenance mismatch is refused, |
Note
Without uow import is not atomic across records and assumes the
destination is not under concurrent write for this model. The
recovery path for an interrupted run is to re-run with
on_conflict="overwrite".
Example
with open("memories.jsonl") as fh: report = Memory.import_records(fh, on_conflict="overwrite") print(report.summary())
Source code in src/popoto/transfer/import_.py
810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 | |