Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.032494258s trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:06:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.032523511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.00075931s trace: https://jaeger.dev.storj.io/trace/ea7a752bc5b1f00 FAIL (2025-12-03 03:05:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000797704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.830999411s trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.831026489s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 426.856533ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:04:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 426.880919ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 547.251058ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:03:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 547.269583ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.034604628s trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.034632408s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 468.831419ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:29:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.852789ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 441.863828ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:28:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 441.889902ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.XX.........................................................] duration: 461.817402ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:27:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.960672ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..XX........................................................] duration: 459.438403ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.467246ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...XX.......................................................] duration: 491.305491ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:25:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.328629ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....XX......................................................] duration: 437.632364ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:24:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.883201ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....XX.....................................................] duration: 442.640318ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:24:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.681191ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......XX....................................................] duration: 425.865728ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.934846ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......XX...................................................] duration: 436.558217ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:22:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 436.58292ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........XX..................................................] duration: 425.891947ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:21:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.942818ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........XX.................................................] duration: 445.495338ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:21:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.55539ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........XX................................................] duration: 503.136814ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.1583ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........XX...............................................] duration: 430.747295ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:20:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.769621ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............XX..............................................] duration: 399.169641ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:19:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 399.202396ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............XX.............................................] duration: 442.016203ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.040789ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............XX............................................] duration: 411.634885ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.660081ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............XX...........................................] duration: 417.845333ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:17:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.880725ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................XX..........................................] duration: 396.259622ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.283137ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................XX.........................................] duration: 661.878945ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:15:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 661.914305ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................XX........................................] duration: 484.000742ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:14:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.046405ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................XX.......................................] duration: 432.23123ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:14:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 432.25373ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................XX......................................] duration: 407.463368ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 407.50715ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................XX.....................................] duration: 417.963498ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.990814ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................XX....................................] duration: 615.692714ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:12:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 615.793249ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................XX...................................] duration: 418.792821ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:12:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 418.821839ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................XX..................................] duration: 451.653481ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:11:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.695967ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................XX.................................] duration: 427.796106ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 427.837475ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................XX................................] duration: 411.944934ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:10:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.979835ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................XX...............................] duration: 597.351452ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:09:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 597.380221ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................XX..............................] duration: 682.734292ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:08:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 682.758296ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................XX.............................] duration: 480.534771ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:08:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 480.613435ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................XX............................] duration: 670.488577ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 670.519422ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................XX...........................] duration: 454.532115ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.622162ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................XX..........................] duration: 415.717714ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:06:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.782906ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................XX.........................] duration: 453.746379ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:05:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.76759ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................XX........................] duration: 443.575781ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:04:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 443.610565ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................XX.......................] duration: 446.041274ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:03:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.104659ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................XX......................] duration: 783.035311ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:02:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 783.058574ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration: 404.694941ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:01:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 404.720584ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 411.801202ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:01:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.823711ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 479.111646ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:00:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.14551ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................XX..................] duration: 594.822374ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 23:00:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.95663ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................XX.................] duration: 481.603971ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 22:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.701064ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 513.502665ms trace: https://jaeger.dev.storj.io/trace/4b96ae6b186d10f9 FAIL (2025-12-02 22:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.532788ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)