Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X.XX........................................................] duration:
661.547369ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:24:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 661.572777ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration:
496.538531ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:24:27 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.567157ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [XXX..........X...X.........X....X....X.......X..............] duration:
1.259605216s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:24:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.259636276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.......] duration:
425.36726ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:24:05 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 425.39972ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
495.607029ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:23:52 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 495.622116ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X..........XX..X....X........X.............................] duration:
1.177891687s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:23:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.177923513s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [XX.XX.......................................................] duration:
666.052035ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:23:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 666.101094ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X......] duration:
517.020813ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:23:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 517.080546ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.XXX..........X...X.........X....X....X.......X.............] duration:
1.221512364s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:23:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.221552617s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [XXX.XX......................................................] duration:
817.019404ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:23:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 817.061121ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.....] duration:
443.613392ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:23:02 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 443.695837ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..XXX..........X...X.........X....X....X.......X............] duration:
1.308739437s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:22:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.308764415s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
558.361172ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:22:42 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 558.412396ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X..........XX..X....X........X............................] duration:
1.113249582s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:22:37 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.113322685s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...XXX..........X...X.........X....X....X.......X...........] duration:
1.235397112s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:22:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.235433158s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
550.551612ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:22:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 550.578995ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [XXXX.XX.....................................................] duration:
660.632408ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:22:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 660.660615ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X..........XX..X....X........X...........................] duration:
1.302241421s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:22:04 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.302282407s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X....] duration:
441.076563ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:21:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 441.107178ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X..........XX..X....X........X..........................] duration:
1.060366335s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:21:48 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.060397924s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [....XXX..........X...X.........X....X....X.......X..........] duration:
1.191198167s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:21:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.191228686s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration:
488.275849ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:21:19 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.327396ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X..........XX..X....X........X.........................] duration:
1.196776219s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:21:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.196809877s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [XXXXX.XX....................................................] duration:
616.615506ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:21:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 616.642393ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration:
546.826641ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:20:57 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 546.878632ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X...] duration:
406.909871ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:20:36 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 406.940641ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....XXX..........X...X.........X....X....X.......X.........] duration:
1.091679184s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:20:33 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.091698333s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [XXXXXX.XX...................................................] duration:
589.274067ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:20:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 589.299906ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration:
726.063279ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:20:12 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 726.343539ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X..........XX..X....X........X........................] duration:
1.326039787s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:20:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.326137195s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration:
529.018316ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:19:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.069742ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X..] duration:
406.251522ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:19:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 406.282342ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.XXXXXX.XX..................................................] duration:
677.549252ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:19:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 677.579225ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.] duration:
426.622723ms trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:19:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 426.64411ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [......XXX..........X...X.........X....X....X.......X........] duration:
1.228025395s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:19:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.228060105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..XXXXXX.XX.................................................] duration:
599.940439ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:19:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 599.968825ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X] duration:
30.02261645s trace:
https://jaeger.dev.storj.io/trace/68ad04ba5c601def
FAIL (2026-03-18 20:19:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload
Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload
Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments...
Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false
Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment...
Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment.
Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece...
Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment
Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments...
Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil>
Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments...
Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.022657321s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X..........XX..X....X........X.......................] duration:
1.142082316s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:19:12 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.142110925s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [X.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration:
536.556345ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:19:02 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 536.585056ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...XXXXXX.XX................................................] duration:
814.732783ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:18:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 814.907708ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......XXX..........X...X.........X....X....X.......X.......] duration:
1.386627156s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:18:25 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.386658874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..] duration:
398.209905ms trace:
https://jaeger.dev.storj.io/trace/4655811ada2b7979
FAIL (2026-03-18 20:18:11 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:16:45.168516210 (297h16m42.765287245s) [upload=332961]: Starting upload
Mar 18 20:16:45.168577895 (297h16m42.765348927s) [upload=332961]: Done starting upload
Mar 18 20:16:45.168711660 (297h16m42.765482684s) [upload=332961]: Uploading segments...
Mar 18 20:16:45.168840198 (297h16m42.765611213s) [upload=332961]: Got next segment. Inline: false
Mar 18 20:16:45.168881241 (297h16m42.765652266s) [upload=332961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:16:45.168886171 (297h16m42.765657197s) [upload=332961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:16:47.196356553 (297h16m44.793127577s) [seg_pos={0 0}, upload=332961]: Begin upload segment...
Mar 18 20:16:47.196451859 (297h16m44.793222892s) [seg_pos={0 0}, upload=332961]: Done begin upload segment.
Mar 18 20:16:47.196741750 (297h16m44.793512778s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.199338232 (297h16m44.796109248s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.200556587 (297h16m44.797327603s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.201992032 (297h16m44.798763048s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.207224672 (297h16m44.803995689s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.208081435 (297h16m44.804852452s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.210256535 (297h16m44.807027558s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.212078721 (297h16m44.808849737s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.214581476 (297h16m44.811352485s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.217434080 (297h16m44.814205104s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.220446152 (297h16m44.817217167s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.222777337 (297h16m44.819548354s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.225963836 (297h16m44.822734955s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.228972751 (297h16m44.825743767s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.232619020 (297h16m44.829390037s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.235321732 (297h16m44.832092748s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.280083394 (297h16m44.876854417s) [upload=332961]: Next returned nil segment
Mar 18 20:16:47.280097207 (297h16m44.876868231s) [upload=332961]: Waiting for error group managing segments...
Mar 18 20:16:47.284956192 (297h16m44.881727226s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.285026311 (297h16m44.881797335s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.285594755 (297h16m44.882365791s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.286555006 (297h16m44.88332603s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.287363557 (297h16m44.884134591s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.294896048 (297h16m44.891667081s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.296893527 (297h16m44.893664551s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.297430780 (297h16m44.894201818s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.299517292 (297h16m44.896288305s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.305052080 (297h16m44.901823106s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.305526292 (297h16m44.902297317s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:17:15.132499335 (297h17m12.72927036s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.133503333 (297h17m12.730274376s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:145.79.227.28:4728): protocol: rpc: dial tcp 145.79.227.28:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.133561110 (297h17m12.730332141s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.134239018 (297h17m12.731010051s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: rpc: dial tcp 145.79.227.43:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.134625412 (297h17m12.731396448s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.135255447 (297h17m12.73202646s) [seg_pos={0 0}, upload=332961]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:17:15.156515485 (297h17m12.753286522s) [upload=332961]: Done uploading segments...
Mar 18 20:17:15.156591674 (297h17m12.753362706s) [upload=332961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 398.289942ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [XX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration:
578.673086ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:18:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 578.707694ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X..........XX..X....X........X......................] duration:
1.211957352s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:18:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.211989783s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [....XXXXXX.XX...............................................] duration:
734.122505ms trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:18:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 734.242098ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X..........XX..X....X........X.....................] duration:
1.288624221s trace:
https://jaeger.dev.storj.io/trace/589419efc52e5e2b
FAIL (2026-03-18 20:17:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload
Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload
Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments...
Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false
Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment...
Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment.
Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece...
Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment
Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments...
Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil>
Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments...
Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.288659398s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [........XXX..........X...X.........X....X....X.......X......] duration:
1.264642295s trace:
https://jaeger.dev.storj.io/trace/24517a7cd40ab76d
FAIL (2026-03-18 20:17:33 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
oad
Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments...
Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false
Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment...
Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment.
Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece...
Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment
Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments...
Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil>
Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments...
Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.264694321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
515.070584ms trace:
https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594
FAIL (2026-03-18 20:17:18 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 515.315481ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X.] duration:
516.227609ms trace:
https://jaeger.dev.storj.io/trace/4655811ada2b7979
FAIL (2026-03-18 20:17:15 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:16:45.168516210 (297h16m42.765287245s) [upload=332961]: Starting upload
Mar 18 20:16:45.168577895 (297h16m42.765348927s) [upload=332961]: Done starting upload
Mar 18 20:16:45.168711660 (297h16m42.765482684s) [upload=332961]: Uploading segments...
Mar 18 20:16:45.168840198 (297h16m42.765611213s) [upload=332961]: Got next segment. Inline: false
Mar 18 20:16:45.168881241 (297h16m42.765652266s) [upload=332961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:16:45.168886171 (297h16m42.765657197s) [upload=332961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:16:47.196356553 (297h16m44.793127577s) [seg_pos={0 0}, upload=332961]: Begin upload segment...
Mar 18 20:16:47.196451859 (297h16m44.793222892s) [seg_pos={0 0}, upload=332961]: Done begin upload segment.
Mar 18 20:16:47.196741750 (297h16m44.793512778s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.199338232 (297h16m44.796109248s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.200556587 (297h16m44.797327603s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.201992032 (297h16m44.798763048s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.207224672 (297h16m44.803995689s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.208081435 (297h16m44.804852452s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.210256535 (297h16m44.807027558s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.212078721 (297h16m44.808849737s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.214581476 (297h16m44.811352485s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.217434080 (297h16m44.814205104s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.220446152 (297h16m44.817217167s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.222777337 (297h16m44.819548354s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.225963836 (297h16m44.822734955s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.228972751 (297h16m44.825743767s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.232619020 (297h16m44.829390037s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.235321732 (297h16m44.832092748s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Uploading piece...
Mar 18 20:16:47.280083394 (297h16m44.876854417s) [upload=332961]: Next returned nil segment
Mar 18 20:16:47.280097207 (297h16m44.876868231s) [upload=332961]: Waiting for error group managing segments...
Mar 18 20:16:47.284956192 (297h16m44.881727226s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.285026311 (297h16m44.881797335s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.285594755 (297h16m44.882365791s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.286555006 (297h16m44.88332603s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.287363557 (297h16m44.884134591s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.294896048 (297h16m44.891667081s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.296893527 (297h16m44.893664551s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.297430780 (297h16m44.894201818s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.299517292 (297h16m44.896288305s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.305052080 (297h16m44.901823106s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:16:47.305526292 (297h16m44.902297317s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil>
Mar 18 20:17:15.132499335 (297h17m12.72927036s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.133503333 (297h17m12.730274376s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:145.79.227.28:4728): protocol: rpc: dial tcp 145.79.227.28:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.133561110 (297h17m12.730332141s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.134239018 (297h17m12.731010051s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: rpc: dial tcp 145.79.227.43:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.134625412 (297h17m12.731396448s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:15.135255447 (297h17m12.73202646s) [seg_pos={0 0}, upload=332961]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:17:15.156515485 (297h17m12.753286522s) [upload=332961]: Done uploading segments...
Mar 18 20:17:15.156591674 (297h17m12.753362706s) [upload=332961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 516.244475ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.....XXXXXX.XX..............................................] duration:
4.658750631s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 20:17:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.658770743s
core
uplink
us-select-1
us1-placement-20
|